Featured post

Linux Internals: How /proc/self/mem writes to unwritable memory


An obscure quirk of the /proc/*/mem pseudofile is its “punch through” semantics. Writes performed through this file will succeed even if the destination virtual memory is marked unwritable. In fact, this behavior is intentional and actively used by projects such as the Julia JIT compiler and rr debugger.

This behavior raises some questions: Is privileged code subject to virtual memory permissions? In general, to what degree can the hardware inhibit kernel memory access?

By exploring these questions1, this article will shed light on the nuanced relationship between an operating system and the hardware it runs on. We’ll examine the constraints the CPU can impose on the kernel, and how the kernel can bypass these constraints.

Continue reading
Featured post

Double fetches, scheduling algorithms, and onion rings

Most people thought I was crazy for doing this, but I spent the last few months of my gap year working as a short order cook at a family-owned fast-food restaurant. (More on this here.) I’m a programmer by trade, so I enjoyed thinking about the restaurant’s systems from a programmer’s point of view. Here’s some thoughts about two such systems.

Continue reading
Featured post

What they don’t tell you about demand paging in school

This post details my adventures with the Linux virtual memory subsystem, and my discovery of a creative way to taunt the OOM (out of memory) killer by accumulating memory in the kernel, rather than in userspace.

Keep reading and you’ll learn:

  • Internal details of the Linux kernel’s demand paging implementation
  • How to exploit virtual memory to implement highly efficient sparse data structures
  • What page tables are and how to calculate the memory overhead incurred by them
  • A cute way to get killed by the OOM killer while appearing to consume very little memory (great for parties)

Note: Victor Michel wrote a great follow up to this post here.

Continue reading
Featured post

How setjmp and longjmp work (2016)

Pretty recently I learned about setjmp() and longjmp(). They’re a neat pair of libc functions which allow you to save your program’s current execution context and resume it at an arbitrary point in the future (with some caveats1). If you’re wondering why this is particularly useful, to quote the manpage, one of their main use cases is “…for dealing with errors and interrupts encountered in a low-level subroutine of a program.” These functions can be used for more sophisticated error handling than simple error code return values.

I was curious how these functions worked, so I decided to take a look at musl libc’s implementation for x86. First, I’ll explain their interfaces and show an example usage program. Next, since this post isn’t aimed at the assembly wizard, I’ll cover some basics of x86 and Linux calling convention to provide some required background knowledge. Lastly, I’ll walk through the source, line by line.

Continue reading

gardenOS Update 1

(This is a random collection of thoughts around my new operating systems project, “gardenOS”.)

What’s up with the “gardening” terminology?

This is a phrase that I feel perfectly describes the ethos of this project. In the same way people have gardens as calm places to express themselves, learn things, and have fun doing work, I want to create an analogous place for exploring my interest in operating systems.

Some key tenants of the approach:

  • No stress
  • Have fun
  • Pursue whatever interests you

I don’t have particularly strong OSdev skills at the moment, so I’m especially focusing on doing small, easy tasks, such as cleaning up the build system. I do this in the same way you might spend an afternoon picking weeds in your little garden. It’s easy, well understood, not too complicated, no large decisions to be made — and it concretely improves the project. It’s a concrete win you can lock in, in a fixed amount of time, with fairly little work.

Disclaimer: I’m almost apprehensive to even give this project a name

I’m worried that even naming this project (“gardenOS”) will put too much pressure on me. I am deeply aware that OS’s take monumental amounts of time and energy to even get to basic states. And at my current rate (~2 hours a week), it’s unlikely we will get to even basic levels soon.

I wasn’t expecting things to get this philosophical this quickly. My focus above all, is to have fun, learn things, and make some small progress each week in the stream. Each week where I do a stream or do some work, any any of that happens, is a win.

I explicitly hold very few expectations around a future “goal” of the project or where I want it to end up. I just want to have fun and learn things about operating systems.

The loose vision I have is to create a minimal OS for play and experimentation. It should be a high quality codebase, and I should work on it as if I wanted to present my best self as an aspiring pro systems programmer.

The ethos of the project

Even though the project is in a maximally nascent stage, I already feel a certain ethos evolving. In the project, we emphasize:

  • Relaxed, casual, kind attitude
  • Learning mindset
  • Ambition to use programming best practices, and aspiring to become pro systems programmers
  • Portray an accurate “slice of life” of systems programming. Meaning: Show the day to day, which is not always thrilling tasks (like adding a syscall), but simply just working on the build system or refactoring some code.

My experience after doing four streams

I’ve done four streams so far (2 live, 2 recorded). My big worries are that I’ll run out of stuff to do and have to scramble to find something to do, live.

This has never happened — I’m always surprised at the adventures “we” find ourselves going on

The use of “we”

The project is just me at the moment, but there’s something about using “we” to describe it that makes me feel good.

It’s not just to make myself seem more impressive, or feel less alone. When I use it, I think of the handful of enthusiastic people that have joined me in the live streams, or left comments with questions or encouragement.

Even in these earliest of days, there is some tiny, microscopic community feel forming. So when I say “we”, I speak for this community.

The astonishing cost/benefit asymmetry of a four-day work week

Work update: I reduced to working 4 days/week and the cost/benefit asymmetry is astonishing.

Just one extra free day might not sound like much, but I feel like I gain >100% more high quality free time (Friday off is even better than Saturday; Sunday is not high quality free time for me – too much adulting to do).

And I lose only 20% of my productive work capacity (Well a bit more; Friday would be a more productive day than average for me b/c it’s quieter & less meetings).

I’ve constantly felt squashed the last few years, but always convinced myself it was a me problem, rather than a possibility that even “normal” working hours didn’t actually leave me with enough free time for everything I had to do*. (Maybe some of both)

But my energy and mood are way better than in a long time, so maybe it goes to show that the latter was the case, and one extra day can go a long way. (But that’s less surprising when you frame it as 100% more time).

I do need to be a bit more conscious of how I use my work week, and I have noticed a tendency to try to fit 5 days of work into 4… but overall it’s going well. I’m curious if I end up filling up the extra personal capacity and end up just as stressed, but I somehow doubt that will happen.

*You might ask, well why are you so busy anyway? Are you just piling on voluntary responsibilities?

I’ve thought about this at length and I think my answer is glibly, “expat life”.

Tip: When you’re learning a new programming language, look up prominent open source projects and copy their style

Tip: When you’re learning a new programming language, look up prominent open source projects and copy their style.

Aside from the core language, there are many conventions & little details to learn: naming (variables, classes, files), file structuring, literal code formatting

These are things few blogs talk about because it’s highly opinionated. But nevertheless when you’re learning, you’ll benefit from at least some reference for these.

Find a few “professional” open source projects and browse to see what various interpretations of “professional style” are. Then pick one you like most.

Be careful of picking projects that are too old — they might use older style for consistency with legacy, even though they might ideally wish to modernize it.

And ideally pick projects whose contributors are experienced engineers who work on it full-time. Since they “live in” the codebase, they’re less likely to tolerate sloppiness – or are at least more invested in cleaning it up.

The last idea is influenced by @awesomekling, who talks about similar things in his classic “My top advice for programmers looking to improve” car-talk video =]

Even if people already know the plot, they appreciate the way you tell the story

This is something my former colleague, Trent Brunson, told me on Twitter once that I’ll never forget. I tweeted earnestly how there were many things I wanted to share, but was concerned that everyone else knows these things already.

Trent’s (now-deleted) reply was:

I can’t think of a single time where I haven’t learned something from you. Keep sharing! Even if people already know the plot, they appreciate the way you tell the story.

So go forth, share freely, and don’t worry if the topic has already been covered, or if people are already familiar with what you’d like to share.

Accept the constraint and deal with the consequences

I deal a lot with decision paralysis and this has been a helpful mental tool. Decisions I might get stuck on:

  • Which product do I buy of these options
  • Which apartment do I move into
  • Which travel itinerary do I book

Decisions are effectively constraints on your life path. At one point, you have N options open to you, but after the (permanent) decision, you now have only one, the one you chose.

The paralysis comes from worrying about making the wrong decision and suffering because of it. But at a certain point, the decision process saturates. You run out of time and energy, and spending more of them doesn’t create a better decision. (It might worsen it in fact as you second-guess a previously good decision.) But even at this saturation point, you still might be unsure what to do.

Here, I find that it helps to just:

  • Recognize that moving forward is more important than making the “best” decision
  • Pick a promising option and accept that I’ll deal with whatever future consequences
  • Move on

Creative potential energy

Creative potential is to potential energy as creative output is to kinetic energy.

I feel strongly that everyone has potential in them to be creative in some form — “creative potential”. When someone creates something, the potential is released into some kind of output — “creative output”. (This applies even to ephemeral creativity like a live performance.)

Just like how electric potential energy (voltage) is converted to tremendous kinetic energy (current) with lightning, or via a short in a circuit, creative potential can be violently converted into creative output when the conditions are right (a “channel”). Another name for this process is “creative flow”.

The trick is finding the right conditions for a person to unleash their creative potential. The space of environmental conditions is vast with infinite parameters, including:

  • Art form
  • Instruments available
  • Time & place
  • Solo or with others

While I firmly believe people can discover their medium at any age, I think it’s important to focus on finding at least one pre-adulthood. Unfortunately after that point I think people lose confidence in themselves and resign themselves as “uncreative”, which can be a tough hurdle to get over.

I’m happy to say I’ve personally, finally found a channel for my writing practice, right here on this blog. More on that here.

Unpublished content is like cash under your mattress

I think a good reason to publish art sooner than later, and to default to creating in public, is so your art can start accruing interest as soon as possible. (“Art” used loosely; think content & projects also).

If you make art but only ever keep it private, there is a zero chance of it ever being discovered or enjoyed. In that sense, there is some wasted potential, like cash in the mattress.

But if the art is published (not necessarily promoted), there’s a non-zero chance of it being discovered and enjoyed by other people (potentially many others) with no work on your part. That’s what I mean by interest — some potential upside, at no cost to you.

I’ve been practicing this with this very blog. I write a lot and tell no one because that feels good at the moment. However, I already did spot some interest accrued: The “Nixers” newsletter #221 included a link to a recent post where I did zero promotion. Thank you!

P.S. Another advantage is that the work is readily available to be shared if you ever choose to at any point in the future — no need to dig it up and upload it.

Learn to build your own tools

It’s worth it to learn to build your own tools. If you don’t, the only tools you’ll be able to use are those that someone else made, created a company around, then brought to mass market (henceforth, “the song and dance”).

That’s a herculean task, even for software tools which are the easiest to do all that with.

But just because someone didn’t do the song and dance (or failed at any of the infinite points to do so along the way, possibly even due to sheer bad luck) doesn’t mean that tool couldn’t exist today and be actively improving your life, right now.

There is a mind-boggling effort difference between making a prototype product that provides value today, and productionizing that product for the mass market and doing the song and dance.

Prototype products which are only ever used by the creator (i.e. expert user) for utilitarian reasons don’t need to have clean UIs, graceful error handling, or intuitive, consistent, and discoverable UX. They just need to kind of do the job.

The unreal effort required for mass market means that the actual products on the market today are only a tiny subset of the actual products possible in the world now without a technological breakthrough.

You, personally, would probably benefit from some of both sets of products; some that exist today on the market, some that could but no one did yet. Some of the latter might have even been game-changing for you.

But if you know how to build your own tools, you can access that superset of possible products today. Even if it’s just prototype quality tools, that might be good enough.

I’m talking mostly about digital tools but argument applies to anything. What I find especially interesting is how trends in technology continually decrease the effort to build prototype software. This used to require code. Then no-code came around. And finally AI generated code is here. People at large already do this today with tools like Excel, but I wouldn’t be surprised if many more people in the future are using bespoke one-off prototype mobile apps, desktop apps, and web software they’ve built for themselves to solve specific everyday problems.

Personally I do this a bunch already using code and no-code solutions:

And a few no-code apps I’ve built for myself using AppSheet:

  • Personal metrics tracking app
  • Simple personal finance and budgeting app

I’ve also created a custom calendar in Google Sheets which has really improved my life in 2023. Maybe I’ll write about that later.