Are the function signatures correct in the RSA Authentication Agent API documentation?

Question

I have some software which uses the documented API for RSA's Authentication Agent. This is a product which runs as a service on the client machines in a domain, and authenticates users locally by communicating with an "RSA Authentication Manager" installed centrally.

The Authentication Agent's API is publicly documented here: Authentication Agent API 8.1.1 for C Developers Guide. However, the docs seem to be incorrect, and I do not have access to the RSA header files - they are not public; only the PDF documentation is available for download without paying $$ to RSA. If anyone here has access to up to date header files, would you be able to confirm for me whether the documentation is out of date?

The function signatures given in the API docs seem incorrect - in fact, I'm absolutely convinced that they are wrong on x64 machines. For example, the latest PDF documentation shows the following:

int WINAPI AceSetUserData(SDI_HANDLE hdl, unsigned int userData)
int WINAPI AceGetUserData(SDI_HANDLE hdl, unsigned int *pUserData)

The documentation states several times that the "userData" value is a 32-bit quantity, for example in the documentation for AceInit, AceSetUserData, and AceGetUserData. A relevant excerpt from the docs for AceGetUserData:

This function is synchronous and the caller must supply, as the second argument, a pointer to a 32-bit storage area (that is, an unsigned int) into which to copy the user data value.

This is clearly false - from some experimentation, if you pass in a pointer to the center of a buffer filled with 0xff, AceGetUserData is definitely writing out a 64-bit value, not a 32-bit quantity.

My version of aceclnt.dll is 8.1.3.563; the corresponding documentation is labelled "Authentication Agent API 8.1 SP1", and this corresponds to version 7.3.1 of the Authentication Agent itself.

Test code

Full test code given, even though it's not relevant to the problem at all... It's no use to me if someone else runs the test code (I know what it does!), what I need is someone with access to the RSA header files who can confirm the function signatures.

#include <assert.h>
#include <stdlib.h>
#include <stdint.h>

#ifdef WIN32
#include <Windows.h>
#include <tchar.h>
#define SDAPI WINAPI
#else
#define SDAPI
#endif
typedef int SDI_HANDLE;
typedef uint32_t SD_BOOL;
typedef void (SDAPI* AceCallback)(SDI_HANDLE);
#define ACE_SUCCESS                    1
#define ACE_PROCESSING                 150

typedef SD_BOOL (SDAPI* AceInitializeEx_proto)(const char*, char*, uint32_t);
typedef int (SDAPI* AceInit_proto)(SDI_HANDLE*, void*, AceCallback);
typedef int (SDAPI* AceClose_proto)(SDI_HANDLE, AceCallback);

typedef int (SDAPI* AceGetUserData_proto)(SDI_HANDLE, void*);
typedef int (SDAPI* AceSetUserData_proto)(SDI_HANDLE, void*);

struct Api {
  AceInitializeEx_proto AceInitializeEx;
  AceInit_proto AceInit;
  AceClose_proto AceClose;
  AceGetUserData_proto AceGetUserData;
  AceSetUserData_proto AceSetUserData;
} api;

static void api_init(struct Api* api) {
  // All error-checking stripped...
  HMODULE dll = LoadLibrary(_T("aceclnt.dll")); // leak this for the demo
  api->AceInitializeEx = (AceInitializeEx_proto)GetProcAddress(dll, "AceInitializeEx");
  api->AceInit = (AceInit_proto)GetProcAddress(dll, "AceInit");
  api->AceClose = (AceClose_proto)GetProcAddress(dll, "AceClose");
  api->AceGetUserData = (AceGetUserData_proto)GetProcAddress(dll, "AceGetUserData");
  api->AceSetUserData = (AceSetUserData_proto)GetProcAddress(dll, "AceSetUserData");

  int success = api->AceInitializeEx("C:\\my\\conf\\directory", 0, 0);
  assert(success);
}

static void demoFunction(SDI_HANDLE handle) {
  union {
    unsigned char testBuffer[sizeof(void *) * 3];
    void *forceAlignment;
  } u;

  memset(u.testBuffer, 0xA5, sizeof u.testBuffer);

  int err = api.AceGetUserData(handle, (void*)(u.testBuffer + sizeof(void*)));
  assert(err == ACE_SUCCESS);

  fputs("DEBUG: testBuffer =", stderr);
  for (size_t i = 0; i < sizeof(u.testBuffer); i++) {
    if (i % 4 == 0)
      putc(' ', stderr);
    fprintf(stderr, "%02x", u.testBuffer[i]);
  }
  fputc('\n', stderr);
  // Prints:
  // DEBUG: testBuffer = a5a5a5a5 a5a5a5a5 00000000 00000000 a5a5a5a5 a5a5a5a5
  // According to the docs, this should only write out a 32-bit value
}

static void SDAPI demoCallback(SDI_HANDLE h) {
  fprintf(stderr, "Callback invoked, handle = %p\n", (void*)h);
}

int main(int argc, const char** argv)
{
  api_init(&api);
  SDI_HANDLE h;

  int err = api.AceInit(&h, /* contentious argument */ 0, &demoCallback);
  assert(err == ACE_PROCESSING);

  demoFunction(h);

  api.AceClose(h, 0);

  return 0;
}

Answer 1

As you've copied the function/type definitions out of the documentation, you basically don't have and never will have the correct definition for the version of the .dll you're using and could always end up in crashes or worse, undefined behavior.

What you could do is to debug the corresponding .dll:

Do you run Visual Studio? I remember that VS could enter a function call in debug mode and show the assembly, not sure though how it is today. But any disassembler should do the trick. As of x64 ABI register rcx gets the first argument, rdx the second. If the function internally works with the 32bit register names or clears the upper 32bit than you can assume a 32bit integer. If it uses it to load an address (e.g. lea instruction) you could assume a pointer. But as you can see, that's probably not a road you wanna go down...

So what else do you have left?

The document you've linked states a 32-bit and 64-bit library - depending on the platform you use. I guess you use the 64bit lib and that RSA did not update the documentation for this library, but at some point the developers needed to upgrade the library to 64bit.

So think about this way: If you would be the API developer, what is possible to migrate to 64bit and what not. E.g. everything that needs to work across 32/64 implementations (stuff that gets send over the network or stored and shared on disk) cannot be touched. But everything that's local to the instance, can be migrated. As the userData seems to be a runtime thing, it makes sense to support whatever the platform provides: unsigned long on 64bit and unsigned int on 32bit.

You've figured out that userData must be 64 bit. But not because the function writes out a 64bit integer, but because the function sees a 64bit value to start with. As integers are passed by value (I guess in general, but definitely in WINAPI), there's absolutely no chance the function could see the full 64bit value if it would be a 32bit datatype. So most likely, the API developers changed the datatype to unsigned long (in any case to 64bit type).

PS: If you end up putting a pointer into userData, cast the pointer to uintptr_t and store/read that type.

Answer 2

To avoid questions of undefined behavior, please replace your test function with this one, and report what it prints. Please also show us the complete test program, so that people who have access to this library can compile and run it for themselves and tinker with it. I would especially like to see the declarations of the api global and its type, and the code that initializes api, and to know where the type came from (did you make it up as part of this reverse engineering exercise or did you get it from somewhere?)

static void demoFunction(SDI_HANDLE handle) {
  int err = api.AceSetUserData(handle, 0);
  assert(err == ACE_SUCCESS);

  union {
    unsigned char testBuffer[sizeof(void *) * 3];
    void *forceAlignment;
  } u;

  memset(u.testBuffer, 0xA5, sizeof u.testBuffer);

  err = api.AceGetUserData(handle, (void *)(u.testBuffer + sizeof(void*)));
  assert (err == ACE_SUCCESS);

  fputs("DEBUG: testBuffer =", stderr);
  for (size_t i = 0; i < sizeof(u.testBuffer); i++) {
    if (i % 4 == 0)
        putc(' ', stderr);
    printf(stderr, "%02x", u.testBuffer[i]);
  }
  fputc('\n', stderr);
}

(If your hypothesis is correct, the output will be

DEBUG: testBuffer = a5a5a5a5 a5a5a5a5 00000000 00000000 a5a5a5a5 a5a5a5a5

.)