This post details my adventures with the Linux virtual memory subsystem, and my discovery of a creative way to taunt the OOM (out of memory) killer by accumulating memory in the kernel, rather than in userspace.
Keep reading and you’ll learn:
- Internal details of the Linux kernel’s demand paging implementation
- How to exploit virtual memory to implement highly efficient sparse data structures
- What page tables are and how to calculate the memory overhead incurred by them
- A cute way to get killed by the OOM killer while appearing to consume very little memory (great for parties)
Here is everything you need to know to set up a minimal Linux kernel dev environment on Ubuntu 20.04. It works great on small VPS instances, is optimized for a fast development cycle, and allows you to run custom binaries to exercise the specific kernel functionality being developed.
Filmed some screencasts today explaining some interesting behavior with
/proc/self/mem — you can use it to write to unwritable memory (including the text of libc!).
Filmed a screencast exploring some neat mmap behavior — read bits are not enforced for memory mappings. This is because the underlying x86 page table entries have a single bit to toggle between “Read” and “Read/Write”.
With the recent announcement of Apple Silicon (Apple laptops shifting to the 64 bit ARM architecture), it’s a great time to finally learn ARM64!
Pretty recently I learned about
longjmp(). They’re a neat pair of libc functions which allow you to save your program’s current execution context and resume it at an arbitrary point in the future (with some caveats). If you’re wondering why this is particularly useful, to quote the manpage, one of their main use cases is “…for dealing with errors and interrupts encountered in a low-level subroutine of a program.” These functions can be used for more sophisticated error handling than simple error code return values.
I was curious how these functions worked, so I decided to take a look at musl libc’s implementation for x86. First, I’ll explain their interfaces and show an example usage program. Next, since this post isn’t aimed at the assembly wizard, I’ll cover some basics of x86 and Linux calling convention to provide some required background knowledge. Lastly, I’ll walk through the source, line by line.