Step by Step to Run Firecracker without KVM, Thanks to PVM!

A few days ago, I came across PVM through Alex Ellis’s post. What caught my attention was its ability to run Firecracker without KVM, which is typically a requirement. While most cloud providers do support KVM on their instance services, it’s usually limited to specific instance sizes. For instance, AWS requires a metal instance, which can be quite expensive for experimentation purposes. That’s why I’d recommend using DigitalOcean instead - they support KVM on any instance size, which makes testing much more accessible and cost-effective.

You might wonder why bother with PVM when you can simply use DigitalOcean or other cloud providers. For me, it’s about understanding what PVM is and learning from this Linux patch. I won’t dive too deep into the kernel code (as I’m still getting comfortable with it), but I want to show you how we can run Firecracker without KVM. Just keep in mind that this patch is still in the RC (release candidate) phase, so it’s not stable yet - experiment with caution.

Let’s begin! I’m using Hetzner Cloud with a CPX41 instance (8 vCPUs, 16GB RAM). To make the setup process smoother, I’ll share a code snippet that you can add to your cloud user data. This will automatically install all the necessary packages for kernel compilation and Firecracker binary after the server boots up. Here’s the code:

#!/bin/bash

apt update
apt install -y ca-certificates curl net-tools python3-pip make gcc flex bison pkg-config libncurses-dev libelf-dev libssl-dev python3-venv

echo "Install Docker"
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install -y docker-ce docker-ce-cli containerd.io

echo "Install Firecracker"
wget https://github.com/firecracker-microvm/firecracker/releases/download/v1.12.0/firecracker-v1.12.0-x86_64.tgz
tar -xzvf firecracker-v1.12.0-x86_64.tgz
mv release-v1.12.0-x86_64/firecracker-v1.12.0-x86_64 /usr/local/bin/firecracker
rm -rf release-v1.12.0-x86_64

echo "Enable port forwarding"
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p
iptables -P FORWARD ACCEPT

mkdir -p /var/lib/firecracker

After a few minutes, the user data setup will complete. Now we can clone the PVM source code:

git clone https://github.com/virt-pvm/linux.git -b pvm
cd linux

Generate a config.

make olddefconfig
make menuconfig     # select Virtualization > Pagetable-based PV Virtual Machine

Alternatively, you can edit the .config file to add CONFIG_KVM_PVM=m. We also need to update the config by setting CONFIG_SYSTEM_TRUSTED_KEYS and CONFIG_SYSTEM_REVOCATION_KEYS to empty strings before compilation.

Building a custom kernel based on these settings will fail if debian/certs/debian-uefi-certs.pem does not exist. This file can either be obtained from the kernel team’s Git repository and placed into the source tree, or it can be replaced by your own certificate(s), or you’ll have to disable the setting via CONFIG_SYSTEM_TRUSTED_KEYS=”“. Reference: 8.10.4. Compiling and Building the Package

This is necessary because the compilation might fail due to missing certificates. As we’re just experimenting, we can safely disable these security features.

Now it’s time to compile the kernel. This process might take around an hour, so feel free to grab a coffee while waiting:

make -j6

Once the compilation completes, we can install the kernel and its modules:

make modules_install install

Now that we’ve completed the host kernel compilation, we need to generate a guest kernel for Firecracker. The provided vmlinux from the Firecracker documentation won’t work with our PVM module. I’ll guide you through the steps based on the instructions from here:

wget https://raw.githubusercontent.com/virt-pvm/misc/main/pvm-guest.config -O .config
make olddefconfig
make -j6 vmlinux
cp vmlinux /var/lib/firecracker

The guest kernel is ready to use.

Since we have completed the process, the final step is to update our GRUB configuration to use the newly compiled kernel and disable PTI.

Page Table Isolation (PTI) is a Linux kernel feature designed to mitigate the Meltdown security vulnerability, which affects CPU performance and could potentially allow unauthorized access to kernel memory.

sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT="1>2"/' /etc/default/grub
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="\(.*\)"/GRUB_CMDLINE_LINUX_DEFAULT="\1 pti=off"/' /etc/default/grub

Update grub configuration:

grub-mkconfig -o /boot/grub/grub.cfg

That’s it! Time to reboot the system and verify that PVM is working with the new kernel.

After rebooting, run uname -a to confirm you’re using the new kernel version 6.7.0-rc6.

The module isn’t loaded yet, so let’s activate it using modprobe:

modprobe kvm-pvm

Let’s verify that PVM is enabled:

lsmod | grep pvm
kvm_pvm                57344  0
kvm                  1392640  1 kvm_pvm

Perfect! Now for the final test, we’ll run Firecracker. Since there are several steps involved, I’ll use a Python module I wrote:

python3 -m venv venv
source venv/bin/activate
pip3 install firecracker-python

For SSH connectivity, let’s generate a key before building the root filesystem:

ssh-keygen -f id_rsa -N ""

Here’s a minimal Dockerfile that installs just the essential packages:

cat >Dockerfile <<EOF
FROM ubuntu:24.04

COPY id_rsa.pub /root/.ssh/authorized_keys

RUN echo "root:root" | chpasswd
RUN apt update -q \
    && apt install -y \
    systemd \
    systemd-sysv \
    lsb-release \
    kmod \
    init \
    net-tools \
    iputils-ping \
    openssh-server \
    curl \
    vim \
    ca-certificates \
    gpg \
    dnsutils
EOF

Time to build the image and extract the root filesystem from the Docker image to create our rootfs file for Firecracker:

docker build -t ubuntu .
docker create --name extract ubuntu
docker export extract -o /var/lib/firecracker/rootfs.tar
docker rm -f extract
fallocate -l 5G /var/lib/firecracker/rootfs.img
mkfs.ext4 /var/lib/firecracker/rootfs.img
TMP=$(mktemp -d)
mount -o loop /var/lib/firecracker/rootfs.img $TMP
tar -xvf /var/lib/firecracker/rootfs.tar -C $TMP
echo "nameserver 8.8.8.8" > $TMP/etc/resolv.conf
umount $TMP
rm -rf $TMP

For the final step, we’ll run the microVM using a Python module I created to simplify the process without having to run multiple commands:

from firecracker import MicroVM
vm = MicroVM(kernel_file="/var/lib/firecracker/vmlinux", base_rootfs="/var/lib/firecracker/rootfs.img", verbose=True)
vm.create()

Exit from Python interpreter, you will be able to access the microVM from SSH.

ssh -i id_rsa root@172.16.0.2

For more details about using PVM, check out Alex Ellis’s blog post.

What’s next?

Since past few months, I’ve been delve into Firecracker architecture, have a look for some resource which can help me learn better to understand this technology and honestly it pieces hre and there like a puzzle hence I decided to create a list of useful resources here related to microVM which can help everyone finding it easily without type any keyword, and actually I also found some interesting article and put it there. If you found any good materials related to Firecracker or other thant that feel free to add it and appreciated :)

By doing so, Firecracker has become particularly interesting to me for running isolated environments that require kernel interaction. Unlike Docker, which requires careful capability assignment and exposes the host system, Firecracker provides a more secure and isolated environment. It allows me to quickly spin up environments that behave like real VMs, enabling extensive system modifications without the overhead of traditional virtualization. While it’s not suitable for kernel compilation, it’s perfect for exploring kernel capabilities in a controlled environment. If the environment crashes, I can easily recreate it without affecting the host system.

The next thing I might be trying is to run eBPF module as it a bit challanging to make it run inside Firecracker but its possible to use it :) there are still lot of exploration still I’m doing with firecracker, one project catch my eyes is this https://github.com/fadams/firecracker-in-docker, sounds like firecracker-containerd but I feel a bit more simple (not to sure as not having a look further).

What come up in my ideas is, firecracker might not production ready for now what do I mean by that? Unlike we are using container on top of Kubernetes, it easy to manage, so far I couldn’t found if someone create orchestration for firecracker without need of Kubernetes at least similar looks like a Docker, which can we playaround with docker command, so on so forth. I believe due to its behaviour it like a bit difficult to make it fully orchestrator like Kubernetes, networking would be a challenge, volume or etc. But this might the best option for me to use it as testing environment or whether improving CI/CD pipeline speed by create an image ready and firecracker will spin up to run the job, leave the server still clean after deletion or job is done

I think Firecracker isn’t quite ready for production use yet. Here’s why, while containers on Kubernetes are pretty straightforward to manage, I haven’t come across any good orchestration solutions for Firecracker that work without Kubernetes. Sure, there are some projects trying to make it work like Docker, but they’re still in their early days. Building a full orchestration system like Kubernetes for Firecracker would be tricky, you’d need to figure out networking, volume management, and other infrastructure pieces from scratch. That said, I see a lot of potential for using Firecracker in testing environments and CI/CD pipelines. You could prepare your images ahead of time and use Firecracker to spin up isolated environments for your jobs, keeping your server clean and ready for the next jobs.