StackGuard is a compiler extension that enhances the executable code produced by the compiler so that it detects and thwarts buffer-overflow attacks against the stack. The effect is transparent to the normal function of programs. The only way to notice that a program is StackGuard-enhanced is to cause it to execute C statements with undefined behavior: StackGuard-enhanced programs define the behavior of writing to the return address of a function while it is still active.
As described in Section 2, the common form of buffer-overflow attacks are stack smashers. They function by overflowing a buffer that is allocated on the stack, injecting code onto the stack, and changing the return address to point to the injected code. StackGuard thwarts this class of attack by effectively preventing changes to the return address while the function is still active. If the return address cannot be changed, then the attacker has no way of invoking the injected attack code, and the attack method is thwarted.
StackGuard prevents changes to active return addresses by either detecting the change of the return address before the function returns, or by completely preventing the write to the return address. Detecting changes to the return address is a more efficient and portable technique, while preventing the change is more secure. StackGuard supports both techniques, as well as adaptively switching from one mode to the other.
Section 3.1 describes how StackGuard detects changes to the return address. Section 3.2 describes how StackGuard prevents changes to the return address. Section 3.3 discusses motives and methods for adaptively switching between techniques.