x86 Calling Conventions

A calling convention is a way to maintain function calls in the assembly language. To be more accurate, it is a way of maintaining function arguments. Since function calls aren't a built-in feature in assembly as for example in C or C++, we have to work with what we have available.

Some of the calling conventions may push the arguments on the stack, while others may save them into registers, or a combination of these two. Different ways of maintaining function arguments are discussed in this article.

In what each calling convention may differ

  • How are arguments being passed into the function.
  • Who cleans up the stack.
  • How is the return value stored.
  • Function name mangling

    • Caller vs Callee

    A caller is routine that call subroutine. (e.g. main calls printf)

    A callee is subroutine that was called by another routine. (e.g. printf is being called by main)

    __cdecl (C declaration)

    Also called The default calling convention is a default calling convention used within the C programming language.

  • Arguments are being pushed on the stack from right to left.
  • The return value is stored in the EAX register for integer values, for floating point values in ST0 registers.
  • Registers EAX, ECX and EDX are caller-saved. The rest are callee-saved. Meaning that the caller-saved registers shouldn't be changed by the callee.
  • The caller is responsible for cleaning up the stack after the call.
  • Functions have an additional prefix added to their name. '_'. (_func)
    • caller: push ebp ; save old call frame mov ebp, esp ; init new call frame push 3 ; basically mov [ebp-4h], 3 push 2 ; basically mov [ebp-8h], 2 push 1 ; basically mov [ebp-Ch], 1 call callee ; some compilers may refer to this as 'leave' instruction. add esp, Ch ; remove call arguments from the stack mov esp, ebp ; restore the old stack frame. pop ebp ret

    __stdcall (Standard calling convention)

    Standard calling convention is almost the same as __cdecl. This calling convention is used is Win32 API routines.

  • Arguments are being pushed from right to left
  • The stack is cleaned up by callee instead of caller.
  • Return value is stored the same way as with __cdecl.
  • The name of the function has same prefix as __cdecl, except it also contains a postfix '@', followed by a number of bytes (in decimal) representing the size of function arguments. (_func@8)

    • __fastcall (Microsoft calling convention)

  • First two arguments are evaluated from left to right into ECX and EDX. Remaining arguments are pushed onto the stack from right to left.
  • The stack is cleaned up by callee.
  • Return value is stored the same way as with __cdecl.
  • The function name is decorated with a prefix '@' followed by a number of bytes (in decimal) representing the size of function arguments. (func@8)
    • caller: push ebp mov ebp, esp push 3 ; And there comes actual function arguments mov edx, 2 ; the first two arguments stored in EDX and ECX mov ecx, 1 call callee pop ebp ; the stack is cleaned up by the callee ret

    __thiscall (Using 'this' keyword)

    It's used within C++ non-static member functions. Because this convention isn't part of the standard, the implementation may differ on different compilers. There are two main implementations in GCC and MSVC.

    GCC
  • Almost identical to __cdecl. The caller cleans up the stack, parameters are passed from right to left. The difference is the 'this' pointer being passed last, as the first parameter of the function.

    • MSVC
  • The 'this' pointer is being passed in ECX and the callee cleans up the stack, mirroring the __stdcall. If the function has a variable amount of arguments, it's the caller that cleans up the stack.

    • ; GCC caller: push ebp mov ebp, esp push 3 push 2 push this ; this pushed onto the stack call callee add esp, Ch mov esp, ebp pop ebp ret ; MSVC caller: push ebp mov ebp, esp mov ecx, this push 2 push 3 ; this pushed onto the stack call callee pop ebp ; callee cleans up the stack ret