Understanding the Linux Virtual Memory Manager

Description

A programmer wanting to understand the workings of the Linux VM today literally has no choice but to study the kernel source code, line-by-line - an excruciatingly difficult and time-consuming task. This book dedicates itself to explaining, in detail, how the memory manager is implemented in Linux, thereby cutting down the time needed to understand it from many months to mere weeks. The Linux VM is the single most important component of the Linux kernel. The behavior of the VM affects all other kernel subsystems, and has a dramatic impact on overall system performance. This book is unique in that not only does it describe the Linux VM itself in unprecedented detail, it also includes the theoretical foundations for it which is of interest to both developers and students but has been omitted from recent Linux kernel related material. It is split into three parts. The first part begins with an introduction on how to approach reading the code of an open source project. It then provides a detailed description of the VM architecture with the aid of numerous diagrams and call graphs, which is suitable for people who need a clear understanding of how the VM functions. The second part is a detailed line-by-line description of the C source modules (source code commentary) that implement the VM in the Linux 2.4 kernel. The third part describes new features in the upcoming 2.6 kernel.

Back Cover

Finally, a comprehensive guide to the Linux VM!

VM's behavior affects every Linux kernel subsystem and dramatically impacts overall performance. But until now, there was only one way to understand VM: study the poorly documented source one line at a time. Now there's an easier, faster alternative. This book describes VM in unprecedented detail, presenting both theoretical foundations and a line-by-line source code commentary. It systematically covers everything from physical memory description to out-of-memory management. Coverage includes:

• Linux VM 2.4 architecture in depth-with diagrams and call graphs
• Physical memory description, page tables, address spaces, and memory allocation
• High memory, swapping, shared memory, and much more
• Expert guidance for analyzing the code of any open source project
• New Linux 2.6 kernel features in every chapter

Well organized and superbly written, Understanding the Linux Virtual Memory Manager will be indispensable to every kernel programmer and researcher.

CD-ROM INCLUDED

Complete VM Learning Lab! Contains the author's new toolkit for exploring VM, including a browsable version of kernel source, CodeViz call graph generator; and VMRegress for analyzing and benchmarking VM. Also includes all code commentary in HTML, PDF, and plain text formats.

Contents


Preface.


1. INTRODUCTION.
Getting Started. Managing the Source. Browsing the Code. Reading the Code. Submitting Patches.

2. Describing Physical Memory.
Nodes. Zones. Zone Initialization. Initializing mem map. Pages. Mapping Pages to Zones. High Memory. Whats New in 2.6.

3. Page Table Management.
Describing the Page Directory. Describing a Page Table Entry. Using Page Table Entries. Translating and Setting Page Table Entries. Allocating and Freeing Page Tables. Kernel Page Tables. Mapping Addresses to a struct page. Translation Lookaside Bu.er (TLB). Level 1 CPU Cache Management. Whats New in 2.6.

4. Process Address Space.
Linear Address Space. Managing the Address Space. Process Address Space Descriptor. Memory Regions. Exception Handling. Page Faulting. Copying to/from Userspace. Whats New in 2.6.

5. Boot Memory Allocator.
Representing the Boot Map. Initializing the Boot Memory Allocator. Initializing bootmem data. Allocating Memory. Freeing Memory. Retiring the Boot Memory Allocator. Whats New in 2.6.

6. Physical Page Allocation.
Managing Free Blocks. Allocating Pages. Free Pages. Get Free Page (GFP) Flags. Process Flags. Avoiding Fragmentation. Whats New in 2.6.

7. Noncontiguous Memory Allocation.
Describing Virtual Memory Areas. Allocating a Noncontiguous Area. Freeing a Noncontiguous Area. Whats New in 2.6.

8. Slab Allocator.
Caches. Slabs. Objects. Sizes Cache. Per-CPU Object Cache. Slab Allocator Initialization. Interfacing With the Buddy Allocator. Whats New in 2.6.

9. High Memory Management.
Managing the PKMap Address Space. Mapping High Memory Pages. Unmapping Pages. Mapping High Memory Pages Atomically. Bounce Bu.ers. Emergency Pools. Whats New in 2.6.

10. Page Frame Reclamation.
Page Replacement Policy. Page Cache. LRU Lists. Shrinking All Caches. Swapping Out Process Pages. Pageout Daemon (kswapd). Whats New in 2.6.

11. Swap Management.
Describing the Swap Area. Mapping Page Table Entries to Swap Entries. Allocating a Swap Slot. Swap Cache. Reading Pages From Backing Storage. Writing Pages to Backing Storage. Reading/Writing Swap Area Blocks. Activating a Swap Area. Deactivating a Swap Area. Whats New in 2.6.

12. Shared Memory Virtual Filesystem.
Initializing the Virtual Filesystem. Using shmem Functions. Creating Files in tmpfs. Page Faulting Within a Virtual File. File Operations in tmpfs. Inode Operations in tmpfs. Setting Up Shared Regions. System V IPC. Whats New in 2.6.

13. Out Of Memory Management.
Checking Available Memory. Determining OOM Status. Selecting a Process. Killing the Selected Process. Is That It? Whats New in 2.6.

14. The Final Word.


A. Introduction.


B. Describing Physical Memory.
Initializing Zones. Page Operations.

C. Page Table Management.
Page Table Initialization. Page Table Walking.

D. Process Address Space.
Process Memory Descriptors. Creating Memory Regions. Searching Memory Regions. Locking and Unlocking Memory Regions. Page Faulting. Page-Related Disk I/O.

E. Boot Memory Allocator.
Initializing the Boot Memory Allocator. Allocating Memory. Freeing Memory. Retiring the Boot Memory Allocator.

F. Physical Page Allocation.
Allocating Pages. Allocation Helper Functions. Free Pages. Free Helper Functions.

G. Noncontiguous Memory Allocation.
Allocating a Noncontiguous Area. Freeing a Noncontiguous Area.

H. Slab Allocator.
Cache Manipulation. Slabs. Objects. Sizes Cache. Per-CPU Object Cache. Slab Allocator Initialization. Interfacing with the Buddy Allocator.

I. High Memory Mangement.
Mapping High Memory Pages. Mapping High Memory Pages Atomically. Unmapping Pages. Unmapping High Memory Pages Atomically. Bounce Bu.ers. Emergency Pools.

J. Page Frame Reclamation.
Page Cache Operations. LRU List Operations. Re.lling inactive list. Reclaiming Pages From the LRU Lists. Shrinking All Caches. Swapping Out Process Pages. Page Swap Daemon.

K. Swap Management.
Scanning for Free Entries. Swap Cache. Swap Area I/O. Activating a Swap Area. Deactivating a Swap Area.

L. Shared Memory Virtual Filesystem.
Initializing shmfs. Creating Files in tmpfs. File Operations in tmpfs. Inode Operations in tmpfs. Page Faulting Within a Virtual File. Swap Space Interaction. Setting Up Shared Regions. System V IPC.

M. Out of Memory Management.
Determining Available Memory. Detecting and Recovering From OOM.

References.


Code Commentary Index.


Index.


About the Author.

Author

MEL GORMAN specializes in documenting open source software. He holds a MSc in Computer Science from the University of Limerick, Ireland, and has served as an instructor there. He has worked as a system administrator, applications developer, and consultant, and has been researching Linux memory management for more than two years. Currently he is an applications developer with J2EE technologies at IBM, Dublin.