How to set up a minimal Linux kernel dev environment on Ubuntu 20.04

Here is everything you need to know to set up a minimal Linux kernel dev environment on Ubuntu 20.04. It works great on small VPS instances, is optimized for a fast development cycle, and allows you to run custom binaries to exercise the specific kernel functionality being developed.

Step 1: Get a dev machine

I used a Ubuntu 20.04 VPS with 4GB memory and 2 CPUs. The smallest Digital Ocean box (1GB memory) will also work fine, except for cloning the full tree, which can exhaust memory. An easy workaround is to use a shallow clone instead (git clone --depth 1). The actual build will be fine, if a bit slow.

Step 2: Get the kernel source

If you intend to submit a patch, the docs recommend basing your patch off of a recent well-know release point in Linus’ tree.

Patches must be prepared against a specific version of the kernel. As a general rule, a patch should be based on the current mainline as found in Linus’s git tree. When basing on mainline, start with a well-known release point – a stable or -rc release – rather than branching off the mainline at an arbitrary spot.
https://www.kernel.org/doc/html/v4.16/process/5.Posting.html#patch-preparation

Clone Linus’ tree:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Then checkout to a recent release point. You can find one using git tag -l. Once you’ve done that make a new branch for yourself using git checkout -b.

Step 3: Build the kernel

First, install your dependencies¹:

apt install build-essential kernel-package fakeroot libncurses5-dev libssl-dev flex bison libelf-dev

Next, configure the kernel. We will be running the kernel in a minimal QEMU environment, so we can use the default x86_64 config.²

make x86_64_defconfig

This will generate a .config file.

Finally, compile the kernel.

make -j3

On my machine with 4GB memory and 2 cores, this takes 20 minutes. When it’s done, you’ll see something like this:

  ...
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Setup is 14108 bytes (padded to 14336 bytes).
System is 8813 kB
CRC e133314f
Kernel: arch/x86/boot/bzImage is ready  (#8)

Step 4: Get QEMU

Now is a good time to install QEMU, which is an emulator we’ll use to boot our kernel.

apt install qemu qemu-system

Step 5: Build a root filesystem image

If we tried booting the kernel by itself, it would crash when it finds there is no root filesystem to mount. Give it a shot:

qemu-system-x86_64 -kernel arch/x86/boot/bzImage -append "console=ttyS0" -serial stdio -display none

You’ll see something like this:

We ultimately want to boot into a minimal userspace where we can exercise the parts of the kernel we’ll be developing. We can use buildroot to build a root filesystem image, with a minimal, Busybox-based userspace. Then, we can build custom binaries that exercise kernel functionality of interest, and add them to the image using a rootfs overlay. If you don’t need to run custom binaries, you can get away with a ramdisk, which is significantly easier.³

First, clone buildroot.

git clone https://github.com/buildroot/buildroot

Next, prepare a rootfs overlay so we can include additional files in our filesystem image. Again, if you don’t need custom binaries you’re probably better off using a ramdisk than buildroot.

A rootfs overlay is a simply a directory structure which will be copied on top of the plain buildroot image. Here’s what mine looked like:

~/root/
    mybin/
        exhaust

In my case, I wanted to exercise the kernel’s OOM functionality, so I prepared a statically linked binary called exhaust which allocates memory until the OOM killer kills it. Once we configure buildroot with ~/root as the rootfs overlay, the exhaust binary will be available at /mybin/exhaust in the image.

Now we’re ready to configure buildroot. It uses a similar config system that the linux kernel uses. Run make menuconfig and then enable the following configs. (For pictures of this process, check out this post)

Target Options -> Target Architecture -> x86_64
Filesystem images -> ext2/3/4 root file system -> ext4
System configuration -> Root filesystem overlay directories (Set this to your rootfs overlay directory)

Lastly, we can build the root filesystem.

make -j3

Once it completes you should now have a filesystem image located at output/images/rootfs.ext4.

Step 6: Run the kernel

Now that we have our kernel and userspace, we’re ready to run the kernel.

qemu-system-x86_64 -kernel arch/x86/boot/bzImage \
-boot c -m 512 -hda ../buildroot/output/images/rootfs.ext4 \
-append "root=/dev/sda rw console=ttyS0" \
-serial stdio -display none

This will boot the kernel and run the minimal Busybox userspace embedded in the filesystem image. You should see a “Welcome to Buildroot” prompt asking for a login. Type in “root” for the default login. This will drop you into a shell where you can run your binary!

Now you’re ready to do kernel development. You can modify the kernel, run it in an emulated environment, and use custom userspace binaries of your choice to exercise the specific kernel functionality related to your development. Then on to the next step: submitting a patch!

Step 7 (Optional): Run your binary automatically on boot

To further automate the testing process, you can configure init to run your binary automatically after the system has booted.

In your rootfs overlay, create an /etc/init.d directory. This directory contains scripts that init will run. If we manually mount the filesystem, we can take a look at what this contains:

$ mkdir /mnt/myfs
$ mount -t ext4  rootfs.ext2 /mnt/myfs # from inside the images/ dir
$ cd /mnt/myfs/etc/init.d
$ ls -l
total 10
-rwxr-xr-x 1 root 1012 Sep 25 03:42 S01syslogd*
-rwxr-xr-x 1 root 1004 Sep 25 03:42 S02klogd*
-rwxr-xr-x 1 root 2804 Sep 25 03:42 S02sysctl*
-rwxr-xr-x 1 root 1684 Sep 25 03:43 S20urandom*
-rwxr-xr-x 1 root  438 Sep 25 03:42 S40network*
-rwxr-xr-x 1 root  423 Sep 25 03:42 rcK*
-rwxr-xr-x 1 root  408 Sep 25 03:42 rcS*

Then add your own executable script that executes your binary. I called mine S50exhaust.

#!/bin/sh
/mybin/exhaust

Rebuild your filesystem, and that’s it! Now when you boot, your custom binary will automatically run, which can save you typing.

Misc tips

Enabling KVM with QEMU can improve performance (--enable-kvm)
Ccache can significantly help reduce build times (Blog)
Instructions on enabling GDB for kernel debugging can be found here
The Linux kernel does support out of tree builds using make O=build/dir
To install to a custom directory:
- make INSTALL_MOD_PATH=../kernelinstall modules_install
- make INSTALL_PATH=../kernelinstall install

Resources

Learn something new? Let me know!

Did you learn something from this post? I’d love to hear what it was — tweet me @offlinemark!

I also have a low-traffic mailing list if you’d like to hear about new writing. Sign up here:

If this isn’t enough deps, also check here: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
If we were going to install this kernel on our Ubuntu machine, we might want to use the config provided by the Ubuntu distribution as a starting point. This can be found in /boot. make oldconfig will automatically find the existing config and use that as a starting point, but depending on how old that config is, you may be manually prompted for a large number of new configs. make olddefconfig will do the same, but choose default values for new configs, so I recommend using that instead. Be warned that using a distribution config will likely increase build times dramatically due to the additional drivers included.
This is a good guide: http://nickdesaulniers.github.io/blog/2018/10/24/booting-a-custom-linux-kernel-in-qemu-and-debugging-it-with-gdb/

offlinemark

Life, art, and systems programming