1

I'm currently working on a project where there is an executable that needs some C/++ injections to fix code from a DLL and, unfortunately, said exe happens to be compiled with Watcom -- meaning the usual methods of using visual c++ and clang both fail to call the existing code and inject custom functions without breaking parameters (already tried inlining some assembly to pass parameters with stubs, but with over 7000 methods that could arbitrarily be called, it would be a massive headache to maintain).

So, since I'm using Visual Studio 2019 and that has native integration with clang, I was thinking that maybe a good solution would be to use that compiler add implement a custom __watcom calling convention. Now the problem is: how do you even add a custom calling convention? Anybody ever tried doing something similar?

I'm scratching my head to figure out exactly what to change in LLVM or clang.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
Gemini
  • 11
  • 2
  • I don't understand what an "injection" is but if you need to call a set of functions (e.g. those exported by a DLL) that has a different ABI, writing "proxy" code is *way* easier than modifying a compiler. The generation can be automated pretty easily since *all* the thunks will be identical except for the jump to the target function. LLMV is **big**, but [others managed to bend it](https://stackoverflow.com/questions/64601393/clobber-x86-register-by-modifying-llvm-backend). – Margaret Bloom Oct 31 '20 at 20:23
  • An injection is basically a rerouting of code, with a DLL taking over some of it and injecting new routines for bug fixing or behavior changes. Since I would be calling call from the exe while injecting my own with fixes, the calling convention needs to match. – Gemini Oct 31 '20 at 20:41

1 Answers1

1

You will have to make changes in both frontend (Clang) and backend (LLVM).

Changes in the frontend part involves adding support for __watcom keyword, that is making Clang to parse and accept such keyword and somehow propagate it to the backend. One simple approach I see is to

#define __wacom __attribute__((annotate("wacom_calling_convention")))

somewhere in your system headers. This will cause Clang to attach "wacom_calling_convention" metadata to every function marked __wacom.

The backend part is of course more complex. See https://github.com/HikariObfuscator/Core/blob/master/Utils.cpp#L158 for an example how to read annotations on IR level.

Once you find all functions that are marked __wacom, you'll have to change the calling convention for them. See the documentation on that matter. You'll probably have to extend some enum or TableGen file that describes supported calling conventions.

Finally, you'll have make some adjustments on the machine code (MC) layer. I have much less experience with that, so can't give you exact entry point. However, by the time you get to this, you should be able to figure it yourself.

Good luck!

arrowd
  • 33,231
  • 8
  • 79
  • 110