Here is everything you need to know to set up a minimal Linux kernel dev environment on Ubuntu 20.04. It works great on small VPS instances, is optimized for a fast development cycle, and allows you to run custom binaries to exercise the specific kernel functionality being developed.
Step 1: Get a dev machine
I used a Ubuntu 20.04 VPS with 4GB memory and 2 CPUs. The smallest Digital Ocean box (1GB memory) will also work fine, except for cloning the full tree, which can exhaust memory. An easy workaround is to use a shallow clone instead (
git clone --depth 1). The actual build will be fine, if a bit slow.
Step 2: Get the kernel source
If you intend to submit a patch, the docs recommend basing your patch off of a recent well-know release point in Linus’ tree.
Patches must be prepared against a specific version of the kernel. As a general rule, a patch should be based on the current mainline as found in Linus’s git tree. When basing on mainline, start with a well-known release point – a stable or -rc release – rather than branching off the mainline at an arbitrary spot.https://www.kernel.org/doc/html/v4.16/process/5.Posting.html#patch-preparation
Clone Linus’ tree:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Then checkout to a recent release point. You can find one using
git tag -l. Once you’ve done that make a new branch for yourself using
git checkout -b.
Step 3: Build the kernel
First, install your dependencies1:
apt install build-essential kernel-package fakeroot libncurses5-dev libssl-dev flex bison libelf-dev
Next, configure the kernel. We will be running the kernel in a minimal QEMU environment, so we can use the default x86_64 config.2
This will generate a
Finally, compile the kernel.
On my machine with 4GB memory and 2 cores, this takes 20 minutes. When it’s done, you’ll see something like this:
... AS arch/x86/boot/header.o LD arch/x86/boot/setup.elf OBJCOPY arch/x86/boot/setup.bin BUILD arch/x86/boot/bzImage Setup is 14108 bytes (padded to 14336 bytes). System is 8813 kB CRC e133314f Kernel: arch/x86/boot/bzImage is ready (#8)
Step 4: Get QEMU
Now is a good time to install QEMU, which is an emulator we’ll use to boot our kernel.
apt install qemu qemu-system
Step 5: Build a root filesystem image
If we tried booting the kernel by itself, it would crash when it finds there is no root filesystem to mount. Give it a shot:
qemu-system-x86_64 -kernel arch/x86/boot/bzImage -append "console=ttyS0" -serial stdio -display none
You’ll see something like this:
We ultimately want to boot into a minimal userspace where we can exercise the parts of the kernel we’ll be developing. We can use buildroot to build a root filesystem image, with a minimal, Busybox-based userspace. Then, we can build custom binaries that exercise kernel functionality of interest, and add them to the image using a rootfs overlay. If you don’t need to run custom binaries, you can get away with a ramdisk, which is significantly easier.3
First, clone buildroot.
git clone https://github.com/buildroot/buildroot
Next, prepare a rootfs overlay so we can include additional files in our filesystem image. Again, if you don’t need custom binaries you’re probably better off using a ramdisk than buildroot.
A rootfs overlay is a simply a directory structure which will be copied on top of the plain buildroot image. Here’s what mine looked like:
~/root/ mybin/ exhaust
In my case, I wanted to exercise the kernel’s OOM functionality, so I prepared a statically linked binary called
exhaust which allocates memory until the OOM killer kills it. Once we configure buildroot with
~/root as the rootfs overlay, the
exhaust binary will be available at
/mybin/exhaust in the image.
Now we’re ready to configure buildroot. It uses a similar config system that the linux kernel uses. Run
make menuconfig and then enable the following configs. (For pictures of this process, check out this post)
Target Options -> Target Architecture -> x86_64 Filesystem images -> ext2/3/4 root file system -> ext4 System configuration -> Root filesystem overlay directories (Set this to your rootfs overlay directory)
Lastly, we can build the root filesystem.
Once it completes you should now have a filesystem image located at
Step 6: Run the kernel
Now that we have our kernel and userspace, we’re ready to run the kernel.
qemu-system-x86_64 -kernel arch/x86/boot/bzImage \ -boot c -m 512 -hda ../buildroot/output/images/rootfs.ext4 \ -append "root=/dev/sda rw console=ttyS0" \ -serial stdio -display none
This will boot the kernel and run the minimal Busybox userspace embedded in the filesystem image. You should see a “Welcome to Buildroot” prompt asking for a login. Type in “root” for the default login. This will drop you into a shell where you can run your binary!
Now you’re ready to do kernel development. You can modify the kernel, run it in an emulated environment, and use custom userspace binaries of your choice to exercise the specific kernel functionality related to your development. Then on to the next step: submitting a patch!
Step 7 (Optional): Run your binary automatically on boot
To further automate the testing process, you can configure init to run your binary automatically after the system has booted.
In your rootfs overlay, create an
/etc/init.d directory. This directory contains scripts that init will run. If we manually mount the filesystem, we can take a look at what this contains:
$ mkdir /mnt/myfs $ mount -t ext4 rootfs.ext2 /mnt/myfs # from inside the images/ dir $ cd /mnt/myfs/etc/init.d $ ls -l total 10 -rwxr-xr-x 1 root 1012 Sep 25 03:42 S01syslogd* -rwxr-xr-x 1 root 1004 Sep 25 03:42 S02klogd* -rwxr-xr-x 1 root 2804 Sep 25 03:42 S02sysctl* -rwxr-xr-x 1 root 1684 Sep 25 03:43 S20urandom* -rwxr-xr-x 1 root 438 Sep 25 03:42 S40network* -rwxr-xr-x 1 root 423 Sep 25 03:42 rcK* -rwxr-xr-x 1 root 408 Sep 25 03:42 rcS*
Then add your own executable script that executes your binary. I called mine
Rebuild your filesystem, and that’s it! Now when you boot, your custom binary will automatically run, which can save you typing.
- Enabling KVM with QEMU can improve performance (
- Ccache can significantly help reduce build times (Blog)
- Instructions on enabling GDB for kernel debugging can be found here
- The Linux kernel does support out of tree builds using
- To install to a custom directory:
- make INSTALL_MOD_PATH=../kernelinstall modules_install
- make INSTALL_PATH=../kernelinstall install
- https://gist.github.com/chrisdone/02e165a0004be33734ac2334f215380e (ultra minimal config)
Learn something new? Let me know!
Did you learn something from this post? I’d love to hear what it was — tweet me @offlinemark!
I also send out a brief email digest with links to the best writing I do each month. It’s by far the best way to stay up to date:
- If this isn’t enough deps, also check here: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
- If we were going to install this kernel on our Ubuntu machine, we might want to use the config provided by the Ubuntu distribution as a starting point. This can be found in
make oldconfigwill automatically find the existing config and use that as a starting point, but depending on how old that config is, you may be manually prompted for a large number of new configs.
make olddefconfigwill do the same, but choose default values for new configs, so I recommend using that instead. Be warned that using a distribution config will likely increase build times dramatically due to the additional drivers included.
- This is a good guide: http://nickdesaulniers.github.io/blog/2018/10/24/booting-a-custom-linux-kernel-in-qemu-and-debugging-it-with-gdb/
This fully automated setup may also be of interest: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/b3401b61ed052440860adad98eb8e0826cafe9d0#qemu-buildroot-setup-getting-started