Aaron Robinson - 03/22/03
2003 Student Project
Cxbx Xbox Emulator

Cxbx Xbox Emulator Progress Report

What is the purpose of this document?

This document was written in March of 2003 in order to demonstrate progress on Aaron Robinson's student project. This project relates to the emulation of the Microsoft Xbox Game Console. At the time of writing this document, Cxbx was the only Xbox emulator known to be in development. It is also expected to be the first completed Xbox emulator.

What is Cxbx?

Cxbx is an Xbox Emulator. This means that Cxbx is designed to execute software that was designed and compiled for the Xbox console system. The primary use of this tool is to run Xbox games on a PC. While there are many different types of emulation, all share one primary goal in common: To accurately reproduce the behavior of a target system on the host system.

Target System : Xbox Console
Host System : Win32 PC

About Console Emulation...

The console emulation scene consists of gaming enthusiasts from all around the world. There are dozens of consoles that have been successfully emulated, and they are distributed freely all over the internet. Typically, an emulator is written by a software guru who has some genuine interest in the console being emulated, for one reason or another. Console emulators are almost never written for profit, instead as a private project out of sheer curiosity and personal interest. Console emulation has been talked about in many mainstream magazines, including Newsweek, Time Magazine, CNET, and 2600. Emulation websites receive tens of thousands of hits every day, as the scene is still extremely active.

One of the major hurdles that arises when attempting to design and implement a console emulator is the severe lack of information. Where does a person go to gather information about a game console? Unfortunately, companies patent and protect all internal documentation, so most information must be completely reverse engineered. There are also often legal issues when writing an emulator. It is almost always illegal to distribute "ROM" images of console games, so emulators must not provide any help at obtaining these images. On the other hand, it is perfectly legal to distribute the emulator program itself, so long as no copyrighted ROM or other material is included. It is also important to know that it is legal to possess a ROM if you own the game.

There are many reasons why a software programmer is willing to put time and effort into such a program. The main reason is that many people just really like the idea of playing their favorite console games on their PC. There is also the issue of preserving games that may become obsolete and unavailable otherwise. With emulation, you can save a backup copy of the game and play it in the future even if your console has grown old and breaks.

Emulation programming is typically very complex, and is rarely attempted by novice programmers. Emulation of older console systems involves very touchy and precise emulation of not only the theoretical hardware, but the bugs and glitches of the original system. Emulation of modern consoles involves very complicated 3-D graphics, audio, input, and systems programming, on top of the existing complexity of the overall design of an emulator.

Xbox Emulation Theory

The Xbox shares many features of a modern PC. Emulating the Xbox is different from typical emulation in that much of the low level emulation becomes unnecessary. Since the Xbox consists of primarily PC hardware, there are tricks that can be used to take advantage of these similarities. Furthermore, the Xbox Kernel is a modified version of the Windows 2000 Kernel, meaning that High Level Emulation becomes extremely attractive.

The goal of Cxbx is to run Xbox games. Xbox games are designed using a Microsoft tool called the "XDK". These games are programmed using an API that is similar to the Win32 API. Since the Target and Host APIs are similar, function interception and redirection is an attractive emulation method. Cxbx uses this method in part of it's emulation core.

The executable file format on the Xbox is called XBE. This file format was not public information, and had to be reverse engineered. Building off preliminary reverse engineered information from other scene authors, I have released very detailed XBE documentation. This documentation includes source code for parsing XBE files, as well as source code for interpreting internal data structures. Since Cxbx is open source, there is also a wealth of example code available to the general public. In order to execute Xbox code on a PC platform, this file format needed to be converted to the Win32 EXE file format. Cxbx does exactly this.

Cxbx High Level Overview

Cxbx operates in several components. In order to take advantage of object oriented design, and for easy code management, the project is separated into several high level objects. Figure 1 demonstrates an abstract, high level view of the Cxbx object interactions. An explanation of these interactions is detailed below.

Figure 1 : High Level Overview

- Cxbx GUI is responsible for all initial user interactions, excluding controller input emulation and real-time emulation configuration. The Cxbx GUI is designed entirely in low footprint Win32 API. This is one of many design decisions that has allowed the overall size of the Cxbx project to be extremely small. The GUI allows the user to create (open) XBE files, modify if desired, save if desired, and convert to EXE. The user may then execute the converted file, thus beginning the visible emulation stage.

Figure 2 : Cxbx GUI

- Xbe Object is responsible for loading and saving XBE files. The Xbe Object parses through raw binary data, loading all structures and converting data into understandable formats when necessary. The XBE file format documentation details the layout of this structure. It is important to remember that this is a proprietary and undocumented file format, and all information about it was obtained via reverse engineering. The file format conveniently lends itself to be relatively easily converted to EXE format, as XBE is a derivative of the PE (EXE) format.

- Exe Object is responsible for loading and saving EXE files. The Exe Object can be constructed using an XBE file. This is where XBE is converted to EXE and Prolog code is inserted into the EXE. The XBE to EXE conversion process requires detailed knowledge of both file formats. Microsoft does not provide detailed documentation on the PE file format, and no documentation at all on the XBE file format. Because of this, unofficial documentation was used to understand the PE (EXE) file format, and the XBE file format was reverse engineered.

- EmuExe represents the fully converted XBE file. After the conversion process, EmuExe contains all the data from the original Xbe, as well as additional "prolog" code which is used to initialize important emulation components. The conversion process ensures that all code will be loaded into memory at the correct location in virtual memory. When EmuExe is executed by the user (usually through the GUI component), the first piece of code to be executed is the prolog code. The prolog code then immediately transfers control over to the Cxbx Krnl object. Additionally, important Kernel Function Hijacking occurs at this phase. This technique is described later in this document.

- Cxbx Krnl is where most of the real time emulation occurs. When EmuExe transfers control to Cxbx Krnl's "init" function, some very important events occur. First of all, an optional Debug Console is allocated. This aids in development tremendously. The next thing to occur is High Level Function Hijacking. This technique is explained in more detail later in this document. Overall, Cxbx Krnl is responsible for :

  • Initializing the emulated FS structure.
  • Initializing the rendering window, Direct3D and DirectInput
  • Hijacking important High Level Functions
  • Maintaining an emulation state
  • Responding to (and emulating) all redirected Kernel and High Level Function calls
  • Handling all user input, graphics, sound, disk i/o, ...

  • Kernel Function Hijacking

    Xbox software is written on top of a relatively light-weight kernel. In order to emulate the system on a high level, Cxbx must intercept and emulate all interactions between Xbox software and the Xbox Kernel. This is accomplished using Kernel Function Hijacking. Kernel functions are used by Xbox software using a mechanism called the Kernel Thunk Table. This table is used as a function lookup. When Xbox software wishes to execute a kernel function, it performs an indirect call in assembly. One example of an intercepted function would be "PsCreateSystemThreadEx". The thunk number for this particular kernel import happens to be 0xFF. The assembly code for this call would be:

    call [KernelThunkTable + 0xFF]

    Notice that in order to intercept these functions, all thunk numbers and the address of the thunk table must be known. The thunk numbers happened to be quite easy to find, and the thunk table address is encoded inside the XBE file. Cxbx is able to decode the thunk table address and locate the thunk table in virtual memory. Then, Cxbx overwrites all of the pointers in that table to point to interception functions. For PsCreateSystemThread, the shell of this interception function looks something like this:

    // ******************************************************************
    // * 0x00FF - PsCreateSystemThreadEx
    // ******************************************************************
    XBSYSAPI EXPORTNUM(255) NTSTATUS NTAPI xboxkrnl::PsCreateSystemThreadEx
        OUT PHANDLE         ThreadHandle,
        IN  ULONG           ThreadExtraSize,
        IN  ULONG           KernelStackSize,
        IN  ULONG           TlsDataSize,
        OUT PULONG          ThreadId OPTIONAL,
        IN  PVOID           StartContext1,
        IN  PVOID           StartContext2,
        IN  BOOLEAN         CreateSuspended,
        IN  BOOLEAN         DebugStack,
        IN  PKSTART_ROUTINE StartRoutine
        EmuXSwapFS();   // Win2k/XP FS
    	// Emulation code
        EmuXSwapFS();   // Xbox FS
    Figure 3: Example interception function

    The exact actual behavior of this interception function is beyond the scope of this document. Notice that the exact calling conventions and parameters must exist in the interception function, or it will not operate correctly and will result in a crash.

    High Level Function Hijacking

    While similar in some ways to Kernel Function Hijacking, High Level Function Hijacking is an entirely different beast. With HLF Hijacking, we are not lucky enough to have a simple lookup table to intercept. Instead, we must analyze the structure of the XBE itself and locate the function body using carefully generated function signatures. Cxbx accomplishes this daunting task using a special data type and technique I have invented specifically for this job. Consider the following chunk of assembly code :

    sub_114D5                           sub_114D5       proc near               ; DATA XREF: start+5.o
    sub_114D5      8B 0D 18 01 01 00                    mov     ecx, ds:10118h
    sub_114D5+6    A1 08 01 01 00                       mov     eax, ds:10108h
    sub_114D5+B    2B C1                                sub     eax, ecx
    sub_114D5+D    05 00 00 01 00                       add     eax, 10000h
    sub_114D5+12   3B 01                                cmp     eax, [ecx]
    sub_114D5+14   73 02                                jnb     short loc_114ED
    sub_114D5+16   89 01                                mov     [ecx], eax
    sub_114D5+18                        loc_114ED:                              ; CODE XREF: sub_114D5+14.j
    sub_114D5+18   A1 F4 F1 02 00                       mov     eax, ds:dword_2F1F4
    sub_114D5+1D   2B 05 E4 F1 02 00                    sub     eax, ds:dword_2F1E4
    sub_114D5+23   8B 0D E8 F1 02 00                    mov     ecx, ds:dword_2F1E8
    sub_114D5+29   56                                   push    esi
    sub_114D5+2A   8D 44 08 0F                          lea     eax, [eax+ecx+0Fh]
    sub_114D5+2E   57                                   push    edi
    sub_114D5+2F   83 E0 F0                             and     eax, 0FFFFFFF0h
    sub_114D5+32   6A FC                                push    0FFFFFFFCh
    sub_114D5+34   59                                   pop     ecx
    sub_114D5+35   83 C0 04                             add     eax, 4
    sub_114D5+38   A3 14 14 03 00                       mov     dword_31414, eax
    sub_114D5+3D   99                                   cdq
    sub_114D5+3E   F7 F9                                idiv    ecx
    sub_114D5+40   8B 0D EC F1 02 00                    mov     ecx, ds:dword_2F1EC
    sub_114D5+46   33 F6                                xor     esi, esi
    sub_114D5+48   56                                   push    esi
    sub_114D5+49   56                                   push    esi
    sub_114D5+4A   56                                   push    esi
    sub_114D5+4B   68 66 14 01 00                       push    offset loc_11466
    sub_114D5+50   56                                   push    esi
    sub_114D5+51   56                                   push    esi
    sub_114D5+52   89 01                                mov     [ecx], eax
    sub_114D5+54   E8 F2 09 00 00                       call    sub_11F20
    sub_114D5+59   8B F8                                mov     edi, eax
    sub_114D5+5B   3B FE                                cmp     edi, esi
    sub_114D5+5D   75 0A                                jnz     short loc_1153E
    sub_114D5+5F   56                                   push    esi
    sub_114D5+60   6A 01                                push    1
    sub_114D5+62   6A 01                                push    1
    sub_114D5+64   E8 8C 05 00 00                       call    sub_11ACA
    sub_114D5+69                        loc_1153E:                              ; CODE XREF: sub_114D5+5D.j
    sub_114D5+69   57                                   push    edi
    sub_114D5+6A   E8 88 08 00 00                       call    sub_11DCC
    sub_114D5+6F   5F                                   pop     edi
    sub_114D5+70   5E                                   pop     esi
    sub_114D5+71   C3                                   retn
    sub_114D5+71                        sub_114D5       endp

    Figure 4: Red text represents "important" (offset,value) pairs

    It is important to realize that this function, if compiled in a different executable, will not be exactly identical. Absolute and relative addresses are free to change during the linking phase, so we can not compute a simple hash to locate this function in a given piece of code. Instead, a special data structure was designed.

    - Optimized Offset Value Pair Array

    In order to efficiently locate a given chunk of assembly code (i.e. a High Level Function), a database of (offset,value) pairs can be used. Offset represents the offset (in bytes) from the start of the function. Value represents the byte value at that location. With this datatype, we can locate the function by hand, and then write down important (offset,value) pairs. This process is time consuming, but very rewarding. Cxbx is able to successfully (and with no false identifications to date) identify High Level Functions inside an arbitrary XBE file. This is due to the fact that, statistically, carefully chosen (offset,value) pairs are capable of uniquely identifying relocatable code. The likelihood of falsely locating a function body is inversely proportional to the number of pairs combined with the rarity of those pairs.

    If you count the highlighted bytes in Figure 4, you will find 17 bytes of code. With the OOVPA datatype, it is possible to encode this as exactly 2+17+17 = 36 bytes. This is phenomenal in terms of efficiency.

    This process has an interesting side effect of making the programmer intimately familiar with x86 machine code and assembly. For example, glancing at the above code makes it obvious to me that this function is an initialization routine that is spawning off a thread with CreateThread. The routine is also checking the consistency of some values within the XBE header.

    - Redirection

    Now that we have a mechanism with which to locate HLFs, it is a relatively simple task to redirect them. Since the original function will no longer be used, it is safe to overwrite the existing function body with a small piece of code that simply jumps to the interception function. Now, it is a matter of emulating these high level functions accurately. Once again, the behavior of these interception functions is outside of the scope of this document.