Author Archives: Mark

About Mark

Ten years into my journey towards becoming a pro systems programmer, sharing what I learn along the way. Also on Twitter: @offlinemark.

If you're reading this, I'd love to meet you. Please email mark@offlinemark.com and introduce yourself!

75 lessons on life, art, and making things happen

Inspired by Naval Ravikant, when I learn life lessons, I try to compress them into a short phrase so I remember the lesson better. Here are 75 of my personal learnings:

  1. Your lowest points might be your greatest opportunities in disguise.
  2. All truly incredible outcomes start as “crazy” ideas.
  3. If believing everything happens for a reason makes life better, believe it.
  4. Only keep tense what absolutely must be. Relax everything else.
  5. Before they call you visionary, they call you weird.
  6. Everything useful in the world was created by someone who cared enough to push it into reality.
  7. Just because all your friends do something, doesn’t mean you should.
  8. Just because all your friends don’t do something, doesn’t mean you shouldn’t.
  9. Mix your interests to find your area of world-class potential.
  10. World-class expertise is more attainable than you think.
  11. Zoom in unusually far and narrow on anything, and you’ll see things no one has seen before.
  12. Good ideas aren’t enough – they need to look incredible.
  13. It’s easier to get a good deal if you have cash in hand, exact change, arm extended.
  14. Be able to distinguish investments that look like luxuries.
  15. The true cost of things: (Price Paid – Price Sold For) / (# of Times Used).
  16. Invest aggressively in tools used daily.
  17. Money is the worst form of capital. Prefer health, relationships, knowledge, experience.
  18. Half the battle of making great art is knowing the tools to use.
  19. People will tell you the tools they use, if you ask nicely.
  20. Investing aggressively in the right tools will save money in the long run.
  21. When beginning an art form, try many styles, share, and see what works.
  22. When you find what works, stop exploring. Create in that style until you get tired.
  23. Repeat.
  24. New hobbies can have defined, planned lifetimes.
  25. But previous pursuits do remain part of your identity.
  26. Everything you make builds toward your global body of work.
  27. Your global body of work is a ball of dry kindling, waiting for a spark.
  28. The bigger the ball of kindling, the bigger the flame.
  29. The spark might come soon, in decades, or never.
  30. Being public and meeting many people reduces the risk of the latter.
  31. You don’t need to be a world expert to generate novelty.
  32. Remixing is easier than synthesizing from scratch to generate novelty.
  33. The paradox of art: creative decisions lead to different ends. There is no best end, but some are better than others.
  34. Your life is a decades-long performance art project.
  35. A master chef can answer not only the “right” way to make rice, but also: “What if we use half the water? Twice as much? Half the heat?” – because she’s tried.
  36. Everything good in life comes from people.
  37. Find a community where it’s normal to do the things you aspire to do.
  38. Buy your way in if that’s the easiest way.
  39. Cold email or DM people with gratitude and one hyper-specific question.
  40. Don’t assume you’ll be ignored. Test it.
  41. Lack of reply = Test to see how serious you are.
  42. Don’t rely on your memory for following up. Have a system.
  43. Don’t rely on your memory, in general. Have a system.
  44. Mentorship begins the moment they reply.
  45. Finding mentorship is about making yourself an attractive investment.
  46. You’re not a nobody; you’re a rocket on the launch pad.
  47. Show proof of work to de-risk yourself as a mentee.
  48. Go out of your way to travel to where your mentors live.
  49. Some seeds take years to sprout, but bear the most incredible fruit.
  50. Buying something from them is a way to get closer to a potential mentor.
  51. Being in need is a great way to start conversations with strangers.
  52. You can invent small needs on a moment’s notice, anywhere.
  53. For example, simply needing a recommendation.
  54. Compliments are a great way to start conversations with strangers.
  55. You can take actions that make it easier for strangers to start conversations with you, like wearing interesting clothes.
  56. When surrounded by strangers, gravitate toward who shows you warmth.
  57. Mingling is easier when you’re early to an event.
  58. The transition from stranger to friend can happen in seconds.
  59. The connection isn’t crystallized until you’ve followed up later online.
  60. Reach out to everyone on the internet whose work you admire.
  61. Move from email to text message to deepen relationships.
  62. You’re not competing against the best – only those who show up.
  63. Any great pursuit is a marathon. Learn the art of long-term consistency.
  64. Genuine passion = endurance.
  65. Copycats will have weak endurance.
  66. You can often bypass bureaucracy by showing up in person, early.
  67. Do things that terrify you.
  68. Sometimes impossible decisions solve themselves with time.
  69. Focus less on winning; focus more on not losing. (Warren Buffett)
  70. Don’t be afraid to exploit your unfair advantages.
  71. Have a personal agenda.
  72. When no one has a strong opinion, that’s an opportunity to advance your agenda, if you wish.
  73. “A healthy man wants a thousand things. A sick man wants one.”
  74. The only competition is to know yourself as fully as possible, and act with maximum authenticity towards that truth.
  75. Remember: Millions would switch lives with you in a heartbeat, and readily inherit every single one of your problems.

How to level up your life

Every time I’ve leveled up my life, it’s been because of the people I surrounded myself with, who helped pull me in the direction I wanted to go.

I’ve done this four times in the worlds of:

  • Heavy metal music
  • Electronic music
  • Cybersecurity
  • Audio software

And I’m currently doing it to learn operating systems development.

By the time I was 16, I had released two heavy metal albums on the internet. A large reason why this happened was because I surrounded myself online with a community of people who really cared about this.

In these communities, it was completely normal to be recording your own instrumental heavy metal music, and releasing it every 6-12 months.

Imagine a real-life party for this kind of person. You walk in the room, and if you’re not personally making and releasing your own instrumental heavy metal music online, you’re going to be a bit of the odd one out.

You’re doing to do one of two things. Either, you’ll leave the room, because it’s not the room for you
 Or, if you choose to keep hanging out with these people, you’ll probably start making some music.

Working at Ableton has probably been the best example of this in my life. It was one of the hardest rooms to get into, but the learning on the other side has been incredible.

I’ve been able to work with masters of the craft, who have been doing this for 20+ years. And because I’m on the same team as them, they’re incentivized to pull me up to the level I need to be at to work alongside them.

The point is: You need to find alignment between:

  • the things you care about, your passions, what you want
  • the spaces, rooms, and people you’re surrounding yourself with
  • and the natural direction those rooms are going to pull you in.

Exploit your unfair advantages

My YouTube channel recently crossed 10,000 subscribers, and I’ve done this by exploiting an intersection of three of my unique strengths:

  • Systems programming
  • Not being camera shy
  • Discipline & Consistency

I’m not world class in any of these by themselves, but the combination is a bit more rare and helps me to stand out.

I’m definitely not the best programmer in the world.

I’m also definitely not the most charismatic person in the world. But the bar is pretty low for programmers, especially in my niche of systems programming. I’m a lot less camera shy than most programmers I know.

I’m also not the most consistent person, but I’ve been able to sustain a pace of one livestream per week for about two years.

The end result is that I don’t really have competitors. 95% of the people with the technical skill set that I do have no interest in making content or putting themselves out there online. The remaining 5% either don’t quite have the skill set, or don’t quite have the consistency and burn out.

—

Everyone has unfair advantages relative to the other players in the field.

  • Maybe you have a natural inclination for [thing]?
  • Maybe you’re young and beautiful?
  • Maybe you’re experienced and wise?
  • Maybe you have a lot of energy?
  • Maybe you’re calm and comforting?
  • Maybe you have a nice voice?
  • Maybe you’re really tall or strong?
  • Maybe you’re a man in a female-dominated field?
  • Maybe you’re a woman in a male-dominated field?
  • Maybe you’re not shy?
  • Maybe you can hyper-focus so intensely?
  • Maybe you find talking to people effortless?
  • Maybe you have a lot of time?
  • Maybe you have a lot of money?
  • Maybe you’re resourceful under constraints?

Exploiting your unfair advantages is nothing to be guilty for, once you realize that everyone has them.

Doing things in the world is hard enough as it. You can choose to attempt it without exploiting your strengths, but just know you’re playing on extra hard mode.

osdev journal: bootloaders and booting (grub, multiboot, limine, BIOS, EFI)

Here’s my rough lab notes from what I learned during weeks 69-73 of streaming where I did my “boot tour” and ported JOS to a variety of boot methods.


JOS originally used a custom i386 BIOS bootloader. This is a classic approach for osdev: up to 512 bytes of hand written 16 bit real mode assembly packed into the first sector of a disk.

I wanted to move away from this though — I had the sense that using a real third party bootloader was the more professional way to go about this.

Grub

First I ported to Grub, which is a widely used, standard bootloader on Linux systems.

This requires integrating the OS with the Multiboot standard. Grub is actually designed to simply be a reference implementation of a generic boot protocol, called Multiboot. The goal is to allow different implementations of bootloaders and operating systems to all transparently interoperate with each other, as opposed to the specific bootloaders made for each OS which was common at the time of its development.

(Turns out Multiboot never really took off. Linux and BSDs already had bootloaders and boot protocols and never ported to use Multiboot. Grub supports them via implementing their specific boot protocols in addition to Multiboot. I’m not sure any mainstream OS is natively using Multiboot. Probably mostly hobby os projects.)

This integration looks like:

  • Adding a Multiboot header
  • Optionally making use of an info struct pointer in EBX

The Multiboot header is interesting. Multiboot was designed to be binary format agnostic. While there is native ELF support, OS’s need to advertise that they are Multiboot compatible by including magic bytes in the first few KB of their binary, along with some other metadata (e.g. about architecture). The multiboot conforming boot loader will scan for this header. Exotic binary formats can add basic metadata about what load address they need and have a basic form of loading be done (probably just memcpying the entire OS binary into place. The OS might need to load itself further from there if it has non-contiguous segments.)

Then, for Multiboot v1, the OS receives a pointer to an info struct in EBX. This contains useful information provided from the bootloader (cli args, memory maps, etc), which is the second major reason to use a third party bootloader.

There are two versions of the Multiboot standard. V1 is largely considered obsolete and deprecated because this method of passing a struct wasn’t extensible in a backward compatible way. An OS coded against a newer version of the struct (which might have grown) would possibly crash if loaded against an older bootloader that only provided a smaller struct (because it might dereference struct offsets that go out of bounds of the given struct).

So the Multiboot V2 standard was developed to fix this. Instead of passing a struct, it uses a TLV format where the OS receives an array of tagged values, and can interpret only those whose tags it’s aware of.

The build process is a bit nicer for Grub also compared with a custom bootloader. Instead of creating a “disk image” by concatenating a 512 byte assembly block, and my kernel, with Grub you can use an actual filesystem.

You simply create a directory with a specific directory structure, then you can use grub-mkrescue to convert that into an .iso file with some type of CD-ROM based filesystem format. (Internally it uses xorriso). You can then pass the .iso to QEMU with -cdrom instead of -drive as I was doing previously.

Limine

Limine is a newer, modern bootloader aimed at hobby OS developers. I tried it out because it’s very popular, which I now think is well deserved. In addition to implementing essentially every boot protocol, it includes its own native boot protocol with some advanced features like automatic SMP setup, which is otherwise fairly involved.

It uses a similar build process to grub-mkrescue with creating a special directory structure and running xorriso to produce an iso.

I integrated against Limine, but kept my OS as Multiboot2 since Limine’s native protocol only supported 64 bit.

BIOS vs UEFI

Everything I’ve mentioned so far has been in the context of legacy BIOS booting.

Even though I ported away from a custom bootloader to these fancy third party ones, I’m still using them in BIOS mode. I don’t know exactly what’s in these .iso files, but that means they must populate the first 512 bytes of the media with their own version of the 16 bit real mode assembly, and bootstrap from there.

But BIOS is basically obsolete — the modern way to boot a PC is UEFI.

The nice thing about integrating against a mature third party bootloader, is it abstracts the low level boot interface for you. So all you need to do is target Grub or Limine, and then you can (nearly) seamlessly boot from either BIOS or UEFI.

It was fairly easy to get this working with Limine, because Limine provides prebuilt UEFI binaries (BOOTIA32.EFI) and has good documentation.

The one tricky thing is that QEMU doesn’t come with UEFI firmware by default, unlike with BIOS (where SeaBIOS is included). So you need to get a copy of OVMF to pass to QEMU to do UEFI boot. (Conveniently there are pre-built OVMF binaries available by the Limine author).

I failed at getting UEFI booting with Grub to work on my macOS based dev setup, because I couldn’t easily find a prebuilt Grub BOOTIA32.EFI. There is a package on apt, but I didn’t have a Linux machine quickly available to investigate if I could extract the file out of that.

Even though UEFI is the more modern standard, I’m proceeding with just using BIOS simply to avoid dealing with the whole OVMF thing.

Comparison table

ProsCons
Custom– No external dependency– More finicky custom code to support, more surface area for bugs
– Doable to get basics working but nontrivial effort required to reimplement more advanced features in Grub/Limine (a boot protocol, cli args, memory map, etc)
– No UEFI support
Grub– Well tested, industrial strength
– Available prebuilt from Homebrew
– Simple build process, just use single i386-grub-mkrescue to create iso
– Difficult to get working in UEFI mode on Mac (difficult to find a prebuilt BOOTIA32.EFI)
Limine– Good documentation
– Easy to get working for both BIOS and UEFI
– Supports Multiboot/Multiboot2. Near drop in replacement for grub
– Can opt into custom boot protocol with advanced features (SMP bringup)
– Not used industrially, mostly for hobby osdev
– Not packaged in Homebrew, requires building driver tool from source (but this is trivial)

osdev journal: Gotchas with cc-runtime/libgcc

libclang_rt/libgcc are compiler runtime support library implementations, which the compiler occasionally emits calls into instead of directly inlining the codegen. Usually this is for software implementations of math operations (division/mod). Generally you’ll need a runtime support library for all but the most trivial projects.

cc-runtime is a utility library for hobby OS developers. It is a standalone version of libclang_rt, which can be included vendored into an OS build.

The main advantage for me is that it lets me use a prebuilt clang from Homebrew. The problem with prebuilt clang from Homebrew, is it doesn’t come with a libclang_rt compiled for i386 (which makes sense, why would it — I’m on an ARM64 Mac).

(This is unlike the prebuilt i386-elf-gcc in Homebrew, which does come with a prebuilt libgcc for i386).

Since it doesn’t come with libclang_rt for i386, my options are:

Option
Keep using libgcc from i386-elf-gcc in HomebrewUndesirable — the goal is to only depend on one toolchain, and here I’d depend on both clang and gcc.
Build clang and libclang_rt from sourceUndesirable — it’s convenient to avoid building the toolchain from source if possible.
Vendor in a libgcc.a binary from https://codeberg.org/osdev/libgcc-binariesUndesirable — vendoring binaries should be a last resort
Use cc-runtimeBest — No vendored binaries, no gcc dependency, no building toolchain from source

However, cc-runtime does have a gotcha. If you’re not careful, you’ll balloon your binary size.

This is because the packed branch of cc-runtime (which is the default and easiest to integrate) packs all the libclang_rt source files into a single C file, which produces a single .o file. So the final .a library has a single object file in it.

This is in contrast to libgcc.a (or a typical compilation of libclang_rt) where the .a library probably contains multiple .o files — one for each .c file.

By default, linkers will optimize and only use any .o files in the .a library that are needed. But since cc-runtime is a single .o file, the whole thing will get included! This means, the binary will potentially include many libclang_rt functions that are unused.

In my case, the size of one of my binaries went from 36k (libgcc) to 56k (cc-runtime, naive).

To work around this, you either need to use the trunk branch of cc-runtime (which doesn’t pack them all into one .c file). This is ~30 .c files and slightly more annoying to integrate into the build system.

Or, you can use some compiler/linker flags to make the linker optimization more granular and work at the function level, instead of the object file level.

Those are:

Compiler flag:

-ffunction-sections -fdata-sections

Linker flag

--gc-sections

With this, my binary size reduced to 47k. So there is still a nontrivial size increase, but the situation is slightly improved.

Ultimately, my preferred solution is the first: to use the trunk branch. The build integration is really not that bad, and the advantage is you don’t need to remember to use the special linker flag, which you’d otherwise need to ensure is in any link command for any binary that links against cc-runtime.

That said, those compiler/linker flags are probably a good idea to use anyway, so the best solution might be to do both.

Idea pools: A simple AI metaphor (WIP)

(A WIP sketch about AI, productivity, tech)


At work, every so often the product teams take a break from normal work and do a ‘hack sprint’ for working on creative, innovative ideas that aren’t necessarily relevant to hot topics in the main work streams.

This time, many of the designers used AI tools to generate code and build prototypes. Normally, they would have required a developer to collaborate with.

In the end, there were simply more hacks done in the end than otherwise would be. So in this local scope, AI didn’t put devs “out of a job” in the hack sprint because designers no longer needed them.

Instead it just allowed the same fixed pool of people to make more things happen, pulling more ideas into reality, from the infinitely deep idea pool, than before.


The “infinitely deep idea pool” is my preferred mental model here.

There’s people on one end, the pool on the other, and the people can pull ideas out of the pool into reality at a fixed rate.

Here’s productivity is defined as “ideas pulled, per person, per second”.

Improvements to tech increase that “idea pull” rate.

People become redundant when technology improves productivity, and the goal is just to maintain the status quo. Then a smaller number of people with higher productivity can pull the same number of ideas as the previously larger team with less productivity.

But often, the goal is not to just maintain the status quo. It’s way too tempting to try to outdo it, and push beyond. We want to pull more ideas out of the pool, which is always possible because the idea pools are infinitely deep.

And if that’s true, then no one becomes redundant — the team could use as much help as it can get to pull more ideas out. (People * Productivity = Ideas Pulled Per Second) This is the phenomenon I observed in the hack sprint.

But that’s an if. Some organizations might be fine to maintain the status quo, or only grow it a small amount, relative to the productivity increase. Then real redundancy is created.

But that’s only in the local scope of that organization. In the global scope, the global idea pool from which we all draw from is infinite — there will always be ideas for someone to pull.


This metaphor can help explain why technological advancements haven’t yielded the relaxation and leisure promised by past futurists. In order to really benefit like this, you need to hold your needs constant (maintain the status quo) while productivity improves. And that’s very difficult to do.

Tips for networking

I’m not a pro, but here’s what I’ve learned along the way:


Randomly add value to peoples’ lives that you want to keep in touch with.

This looks like:

  • Meet interesting people
  • Learn what they care about
  • What out for related things you see
  • Send them their way

These “things” can be serious, like useful tools, apps, or news — or can be silly, like memes.


Simply check in once in a while.


  • People want to help you, but you need to put in the work too
  • Craft good, compelling, detailed requests for help. As opposed to lazy asks.
  • If someone does something nice for you — like making an intro — always follow up with them and let them know how things went.

VIM tips + lab notes

Underrated commands:

z commands

  • z<cr> – redraw with current line at the top
  • zz – redraw with current line at middle

L and H — great for quickly going to the top or bottom of editor and browsing slightly offscreen

Enable VSCode “Editor smooth scrolling”, or find a vim smooth scrolling plugin to make ctrl+d and ctrl+u actually functionality

[[ and ]] — for quickly browsing opening brackets

Diminishing returns of worrying

Writing this just because I’ve never heard anyone talk about it before:

Worrying about things has increasing, diminishing, and negative returns, just like anything else.

The increasing returns are when a bit of worrying about something causes you to prepare for a situation or otherwise act differently in a way that benefits you.

But after a point, the worrying starts to saturate. You’re already aware of the potential problem, and more worrying e.g. doesn’t necessarily help you become more prepared or choose better actions.

Lastly, worrying even more can actively harm you. Maybe it causes undue stress or prompts you to make poor investments, relative to the likelihood of the event you’re worrying about.

So worry, but just enough.

How to be consistent and achieve success

I think the most important part of achieving consistency is detaching yourself from the outcome of each individual work session, whatever that might be. Here are some example ‘work sessions’ from my life:

  • A particular workout
  • Releasing a song
  • Releasing a blog post
  • Doing a stream
  • Making a youtube video

Attaching yourself to the outcome (e.g. number of views) will only set you up for failure, since inevitably one of the work sessions will ‘flop’.

To detach yourself from individual outcomes, you have to love the long-term journey of whatever you’re doing. The absolute most important part is simply being there, day after day, week after week, over a long period of time.

This can be compressed down to “Showing Up = Winning”.

If you can reframe the requirement for “winning” from “getting a lot of views” or “breaking my personal record” to simply “I showed up”, you give yourself a massive psychological advantage.


P.S. One extra tip:

An extra tip for the creatives: Reframe each release as another piece of progress in building your large public body of work. It may not be today, but someday, your large public body of work will make it all happen for you — and every release is a step towards that, no matter how “well” it does.

P.S. another tip

Establish the habit by simply doing the activity at the same time each week/day and scheduling your life around that as much as possible. Ideally find a time slot with the least contention against other things that come up.

For streaming, I found that Sunday afternoons was usually free and didn’t compete too much against other plans.

But the “scheduling your life around it” is where the rubber really meets the road. That’s where you prove to yourself that you consider this a high priority to you by putting your time where your mouth is.