How to learn Assembly

Calling conventions are, well...hopelessly complicated

In a nutshell, when you call a procedure, it's rarely as simple as just executing a CALL statement, executing the procedure code, and then executing a RETN statement. The calling code and the procedure both have to agree on how to do several things:

• Which registers (if any) should be preserved through the procedure?
• How those registers will be preserved. Will the calling code save them before the call and restore them after the return, or will the procedure save them before executing it's own code and restore them before it returns?
• How will parameters be passed? Through registers? Through memory locations? By sticking data on the stack?
• If arguments are passed on the stack, are they pushed from left-to-right or from right-to-left?
• If arguments are passed on the stack, who is responsible for popping them off when the procedure is finished?

I'm sure that's not an all-inclusive list, but basically a calling convention is a set of rules (a protocol, if you will) that answers those questions.

For high-level languages, the calling convention also defines how the function name in the source code will get translated for the export section. This is particularly important for languages like C++ that support function overloading. That is, you have two or more functions declarations in your source code that are represented by the same symbol, but different arguments. Internally, the compiler actually gives each version of the function a unique name according to the name you gave it and the argument list in the function declaration. Then it looks at all the calls in your source code and changes them to the appropriate mangled name. Since the mangled names are what actually go in the export table, the mangling protocol is included in the calling convention.

Probably the most commonly used conventions are:

• STDCALL - the "native" calling mode for Win32 programs. Procedures generated by compilers for most languages in the Windows environment generate their procedures as STDCALL.
• CDECL - used by "C style" procedures. Mainly used for procedures that require a variable number of arguments. In contrast to most other conventions, stack cleanup is performed by the calling code because only the calling code knows how many arguments are actually being passed on the stack.
• FASTCALL - basically the same as STDCALL except that the first two arguments are passed in registers instead of on the stack. Generates slightly faster code because maniuplating registers is faster than PUSH/POP operations. Not really used that much in application programming unless the programmer was a real speed freak.
• THISCALL - Used to call procedures that belong to a C++ object. An offset to the object that the procedure belongs to is passed in ECX.

By the way, I'm just reciting this from my notoriously shakey memory, so it's probable that I've made a mistake somewhere. If I were you, I wouldn't count on it being entirely accurate