This is a general overview of the OeyEnc code. 

To start with, there is no plugin API for Outlook Express or Windows Mail, that I know of, which allows custom message decoding. This is unfortunate and as a result OeyEnc is largely a hack. 

There are 3 main projects to the application. OeyEnc.dll, OeyEnc.exe and CoJack. OeyEnc.dll implements the yEnc decoder and the support code required to make it work with OE. OeyEnc.exe is the loader and implements the code for injecting OeyEnc.dll into the OE/WM process. CoJack is a support library, in source code form, used by both OeyEnc.dll and OeyEnc.exe. It includes functions to hook a procedure and to inject a DLL into running or newly created processes. The remaining 4 projects are BootStrap and OeyEnc.wix which build the installer, ObjEx which is used by CoJack for exporting assembled code to text format, and InetComm which builds an import library for INETCOMM.DLL as its missing from the latest WDK/WSK.

OeyEnc.dll:
OE relies on MIME encodings, which yEnc isn't part of and hooking into that to add yEnc as an option would be pretty complex. Instead it turns out that OE ends up calling a function in INETCOMM.DLL called MimeOleCreateMessage() for every type of message it opens. This function creates an instance of an object which supports the IMimeMessageW interface. IMimeMessageW inherits from a few interfaces such as IPersistStreamInit, IPersistFile and others. Methods on both of these IPersistXxx variants are what end up being called to load the message source. Specifically IPersistStreamInit::Load() and IPersistFile::Load(). 

To override the Load() functions, OeyEnc.dll provides a hook for MimeOleCreateMessage() which creates an object that provides custom implementations of the IPersistXxx::Load() methods, among others. When these methods are called, a translation of yEnc data to a natively supported format is done, before OE has a chance to work with the message. The nice thing about this is that it's a fairly simple idea. The downside to this is that there are a lot of other methods that must also be implemented to provide the rest of the interfaces supported by IMimeMessageW. To make that easier, the interception process also calls the original MimeOleCreateMessage() function to create an object that can be used to provide default implementations of any interface and/or function that doesn't require overloading. This is all tied together using COM aggregation. The hooking is implemented in CoJack. There is a lot of code in MimeMessage.cpp which is mostly support code for getting things to work properly with OE. There is very little documentation for these interfaces, so getting some of it to work is a challenge.

OeyEnc.exe:
Getting OeyEnc.dll into the address space of OE is the job of the loader, OeyEnc.exe. There are two scenarios for this which are addressed in slightly different manners. The first is when OE is already running. For this case, a remote thread is created in the OE process which simply loads OeyEnc.dll. The code for this is in assembly and part of the CoJack project. There are 32-bit and 64-bit versions. As part of the build process, these files are compiled then the binary code is dumped so that it can be included in the CoJack.h file. The ObjEx (Object Exporter) is what exports the binary code. The code is written so that it can be loaded at any address and still function. The second scenario is more complex and addresses loading OeyEnc.dll into a new OE process. In early versions, prior to x64 support, Detours was used for hooking MimeOleCreateMessage() and injecting OeyEnc.dll into an OE process. If there wasn't an instance of OE running, one would be created and the DLL injected after it finished initializing. The problem with this was that OE could call MimeOleCreateMesssage() before there was a chance to inject OeyEnc.dll. To work around this, the process is created in a suspended state. When a process is created in this state, its primary thread has not had a chance to call the entry point in the application. This allows the entry point to be overwritten with code that does something and then forwards to the original entry point. You have to actually overwrite the code though, because Windows has already read the address of the entry point and is set to call it as soon as the primary thread continues. You also cannot use CreateRemoteThread() at this point, because on some versions of Windows the primary thread is the only one that can initialize required system data, so creating another thread before the primary thread has had a chance to do its initialization creates problems. So in CoJack there are also assembly files with code to load the DLL and reset the entry point code to its original state. The loader copies this code to the suspended process, patches the entry point and resumes the process. If all goes well the DLL will be loaded and MimeOleCreateMessage() hooked before OE's original entry point has a chance to run.

The loader also supports launching OE as a specific identity. The user passes the identity name via the command line. The loader uses the Identity API to logon the specified identity and then starts OE. With Windows Mail however, identities were removed. An alternative is to create a separate account on the machine and launch WM as that user. This accomplishes nearly the same thing as identities. This is supported on Windows XP and up because the CredUI API is used to prompt for the password of the account to run as. 

CoJack:
Originally only the first scenario was supported. When the code was ported to 64-bit the Detours library was no longer usable because it only supported 32-bit code. There is a version for 64-bit code now, but it's commercial and expensive. To get around this, custom hooking code was written, which ended up including a disassembler. The reason for the disassembler is in the way the MimeOleCreateMessage() is hooked. In general, you need to copy enough instructions from the hooked function to be able to insert code that jumps to the replacement. Those instructions are important to the original function though, so if you need to call back into it you have to execute those instructions first, then transfer to the instruction that follows those in the original function.  To do this, you need to move compiled instructions to another location where they can be executed, keeping the entire instruction and any offsets in place. For instance, you don't want to move half of the instruction "MOV EAX, DWORD PTR [0x00400128]", or move an instruction that references data relative to its location at point A, as say "A + 48 bytes" to point B without also adjusting the reference which at point B is no longer "+ 48 bytes". To do this, you have to disassemble the code at runtime, which is different between the various versions of Windows.

The disassembler is table based and is not a listing disassembler. The bulk of the CoJack.h file is the instruction tables. Disassembling an instruction stream takes place in _RelocateInstructionStream(). The process is fairly simple. First prefixes are read to determine the appropriate instruction table to use and to set modifiers to be applied to the instruction. Then the instruction is read and its value is used to read the matching entry in the table. The table entry then specifies characteristics of the instruction and how many other bytes to read for the instruction by default, such as immediate bytes (the 8 in "PUSH 8"), or memory references (the 0x00400128 in "MOV EAX, DWORD PTR [0x00400128]"). A previously read prefix might also change how the instruction is read, such as reading a 2 byte immediate instead of a 4 byte one. If the table entry specified that the instruction uses a relative displacement, then the function knows that it has to recalculate the displacement value when it is moved to a different location. When the function moves the entire instruction, it resets the tracking variables (prefixes, etc) and advances to the next instruction. This is done until either a return statement is found, or the minimum number of bytes required to be relocated has been done. This minimum number of bytes will be the size of the JMP instruction that transfers control from the function being hooked to the target. 

The hooking uses the disassembler to copy entire instructions from the function being hooked, to a placeholder that can be used to call back into the funtion being hooked after it has been hooked. It's similar to Detours, except it supports 32 and 64 bit code and doesn't have all the bells and whistles, like transactions. You also cannot remove a hook once installed, though adding that functionality should be trivial. 

The functions which inject DLLs come in 2 flavors. The first, LoadLibraryInProcess() is for injecting a DLL into an already running process. It copies the code generated by the CjThread.asm files to the remote process, sets up a parameter block passed to the function which contains function pointers and the path of the DLL to load, then uses CreateRemoteThread() to run a thread which executes the code copied. The second, ResumeProcessWithModule() is for continuing a process created using the CREATE_SUSPENDED flag, with a DLL injected prior to the process's entry function being called. It copies code generated by the CjStartup.asm files to the remote process, sets up a parameter block passed to the function which contains function pointers, data related to the entry point of the process and the path of the DLL to load, patches the entry point to execute the code copied, then resumes the process. The code copied loads the DLL, resets the entry point patched to its original state and jumps back to the entry point, which now runs as normal. If any of that fails, the process is terminated.