The Linux Virtual Memory System: Security And Buffer Overflows

Explore how modern Linux virtual memory systems defend against buffer overflow attacks through security features like the NX bit, address space layout randomization, and return-oriented programming mitigation. Understand these mechanisms to grasp how operating systems protect user programs and the kernel from malicious exploits.

We'll cover the following...

- Buffer overflow attacks

Probably the biggest difference between modern VM systems (Linux, Solaris, or one of the BSD variants) and ancient ones (VAX/VMS) is the emphasis on security in the modern era. Protection has always been a serious concern for operating systems, but with machines more interconnected than ever, it is no surprise that developers have implemented a variety of defensive countermeasures to halt those wily hackers from gaining control of systems.

Buffer overflow attacks

One major threat is found in buffer overflow attacksSee https://en.wikipedia.org/wiki/Buffer_overflow for some details and links about this topic, including a reference to the famous article by the security hacker Elias Levy, also known as “Aleph One”., which can be used against normal user programs and even the kernel itself. The idea of these attacks is to find a bug in the target system which lets the attacker inject arbitrary data into the target’s address space. Such vulnerabilities sometimes arise because the ...