History of x86

IBCM vs. x86: Registers

Declaring Variables in x86

section .data
a	DB	 	23
b	DW	 	?
c	DD	 	3000
d	DQ	 	-800
x	DD	 	1, 2, 3
y	TIMES 8 DB	0
str	DB	 	'hello', 0
z	TIMES 50 DD	?

Addressing Memory

mov rax, rbx
mov rax, [rbx]
mov [var], rbx
mov rax, [r13 - 4]
mov [rsi + rax], cl
mov rdx, [rsi + 4*rbx]

mov rax, [r11 - rcx]
mov [rax + r5 + rdi], rbx
mov [4*rax + 2*rbx], rcx

A code block in both C/C++ and Assembly

int n = 5;
int i = 1;
int sum = 0;
while (i <= n) {
    sum += i;
    i++;
}

section .data
n	DQ 5
i	DQ 1
sum	DQ 0

section .text
loop:  	mov rcx, [i]
	cmp rcx, [n]
	jg endOfLoop
	add [sum], rcx
	inc qword [i]
	jmp loop
endOfLoop:

Stack Memory Visualization for myFunc

This is just before the call opcode is invoked.

Stack Memory Visualization for myFunc

This is just after the call opcode is invoked.

Callee Rules (Prologue)

3. Save callee-save registers
   • rbx, rbp, r12-r15
   • only need to do this if callee intends to use them, otherwise, no need to save their contents

THEN, perform body of the function

Stack Memory Visualization for myFunc

This is just after the caller invokes the call opcode.

Stack Memory Visualization for myFunc

This is just after the callee invokes the sub rsp, 8 opcode.

Stack Memory Visualization for myFunc

This is after the myFunc() prologue is completed.

Activation Records

Every time a sub-routine is called, a number of things are (potentially) pushed onto the stack: Registers Parameters Local variables Return address All of this is called the activation record (note that caller-saved registers are not shown in this diagram)

Consider this subroutine

void security_hole() {
    char buffer[12];
    scanf ("%s", buffer); // how C handles input
}

The stack looks like (with sizes in parenthesis):

• Addresses increase to the right (the stack grows to the left)
• What happens if the value stored into buffer is 13 bytes long?
• What happens if the value stored into buffer is 20 bytes long?
  • We overwrite the return address!