Calling 🤙 Conventions in x86

In this article I’m giving an overview of different calling conventions with examples.

Intro

Consider the following code snippet:

int mysteriousFunction(int a, int b)
{
  return a + b + c;
}

mysteriousFunction(2,4);
// then the above function is called 

Let’s see the assembly code for each calling convention

CDECL

The output in assemble would be this:

_mysteriousFunction:
push ebp
mov ebp, esp
mov eax, [ebp + 8]
mov edx, [ebp + 12]
add eax, edx
pop ebp
ret
and

;main function
push 4 ; the second argument is pushed first
push 2 ; the first argument is pushed second
call _mysteriousFunction
add esp, 8

As is clearly seen from the above snippet, the arguments to the function are passed in reverse order by means of PUSH instruction (placed on stack). Observe the _mysteriousFunction’s code: values added to eax, but nothing seems to be returning, just ret. That’s because this calling convention passes return values in eax register by default.

Also observe the add esp, 8 at the very end of assembly snippet. This operation cleans the stack. See another article to learn more about how stack and esp work here. So, the stack is cleaned by the caller (main), not the callee (_mysteriousFunction). Why? The [1] article on Wikibooks states:

This allows CDECL functions to have variable-length argument lists (aka variadic functions). For this reason the number of arguments is not appended to the name of the function by the compiler, and the assembler and the linker are therefore unable to determine if an incorrect number of arguments is used. Variadic functions usually have special entry code, generated by the va_start(), va_arg() C pseudo-functions.

By the way, have you notices _ at the beginning of the function name above? This is name decoration. CDECL functions are almost always prepended with an underscore when translated to assembly. But when disassembling, there will be no name decorations.

STDCALL

The one I’m the most familiar with since the times I worked as a malware analyst. And there is a reason for that. Since most of malware is written for Windows machines… This calling convention is sometimes refered to as “WINAPI” and is used almost exclusively by Microsoft. Therefore it’s the standard calling convention for the Win32 API. The best part is that since STDCALL is strictly defined by Microsoft, all compilers that implement it do it the same way.

That’s now look at the assembly output for the same mysteriousFunction:

:_mysteriousFunction@8
push ebp
mov ebp, esp
mov eax, [ebp + 8]
mov edx, [ebp + 12]
add eax, edx
pop ebp
ret 8
and

push 3
push 2
call _mysteriousFunction@8

As you can see, like it is with CDECL, the arguments are pushed onto the stack in reverse order. And it also writes the return value into eax register, but cleaning the stack is now the callee’s responsibility (mysteriousFunction). Therefore, unlike CDECL, variable-length argument lists are not allowed.

FASTCALL

The only place I encountered this calling convention when analysing malicious programs was when I reversed Delphi code. Since the calling convention is not standart for Windows, when openning such a file in IDA one has to change the calling convention manually. Delphi code is quite tedious to analyse and using this calling convention is not easing the task. That’s because it is not completely standard across all compilers.

Let’s now see our mysteriousFunction when compiled with this calling convention flag:

:@mysteriousFunction@8
push ebp
mov ebp, esp
add eax, edx
pop ebp
ret
and

;the calling function
mov eax, 2
mov edx, 3
call @mysteriousFunction@8

The first 2 or 3 4-byte (or smaller) arguments are passed in registers, with the most commonly used registers being edx, eax, and ecx. Additional arguments, or arguments larger than 4-bytes are passed on the stack,also in reverse order like in the two previous examples. The caller is usually responsible for cleaning the stack (should that need arise).

But its name is not a coincidence. FASTCALL is faster 😮. But since this confusing things with arguments, it’s better to use it only when the program has 1, 2, or 3 4-byte arguments and where speed is essential.

Also note @ at the beginning of the function name. It’s FASTCALL’s indicator 👒 and name decoration.

But it’s also used at the end of the function name to indicate the number of arguments passed to the function. But when disassembling, there will be no name decorations.

When gcc compiles with FASTCALL convention or Windows is using it the first two arguments are pushed into ecx and edx, respectively, before pushing any remaining parameters onto the stack.

THISCALL

This calling convention is used by C++ because of non-static class members. Let’s asume that our above mysteriousFunction was a non-static method of some SuspiciousClass and we are instantiating this class to get the method called:

suspiciousClassInstance.mysteriousFunction(a, b, c);

The call to this function would look like this:

mov ecx, ?mysteriousFunction@SuspiciousClass@@QAEHH@Z
push c
push b
push a
call _MyMethod

So, the pointer to the class is passed in ecx and the arguments are pushed in reverse order onto the stack, and the return value is passed in eax as usual. Notice how the object’s name is looking weird. That’s name mangling. There is no universal name mangling algorithm among compilers. To leave some functions as is and not that the compiler do that wicked stuff

developers may put some functions in extern "C" block to protect them from mangling.

Name-mangled functions sometimes include the name of that function’s class and almost always - the number and type of the arguments (so that overloaded functions can be differentiated by the arguments passed to it). But when disassembling, there will be no name decorations and mangling as well unless the function is exported.

System V AMD64 ABI

Used on MacOS, FreeBSD.

:_mysteriousFunction
push rbp
mov rbp, rsp
mov rax, [rbp + 8]
mov rdx, [rbp + 12]
add rax, rdx
pop rbp
retn

mov rdi, 3
mov rsi, 2
call _mysteriousFunction

The below in taken from here and not yet fully digested:

The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, R9 (R10 is used as a static chain pointer in case of nested functions[24]:21), while XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6 and XMM7 are used for the first floating point arguments. Additional arguments are passed on the stack.

Integer return values up to 64 bits in size are stored in RAX while values up to 128 bit are stored in RAX and RDX.

Floating-point return values are similarly stored in XMM0 and XMM1.

The wider YMM and ZMM registers are used for passing and returning wider values in place of XMM when they exist.

If the callee wishes to use registers RBX, RBP, and R12–R15, it must restore their original values before returning control to the caller. All other registers must be saved by the caller if it wishes to preserve their values.

For leaf-node functions (functions which do not call any other function(s)), a 128-byte space is stored just beneath the stack pointer of the function. The space is called the red zone. This zone will not be clobbered by any signal or interrupt handlers. Compilers can thus utilize this zone to save local variables. Compilers may omit some instructions at the starting of the function (adjustment of RSP, RBP) by utilizing this zone. However, other functions may clobber this zone. Therefore, this zone should only be used for leaf-node functions. gcc and clang offer the -mno-red-zone flag to disable red-zone optimizations.

If the callee is a variadic function, then the number of floating point arguments passed to the function in vector registers must be provided by the caller in the AL register.

Unlike the Microsoft calling convention, a shadow space is not provided; on function entry, the return address is adjacent to the seventh integer argument on the stack.

Conclusion

References

[1] Wikibook1

[2] Wikibook2