Part 1: Getting Started with Firecracker
I’ve been exploring an interesting project called Firecracker from AWS, a lightweight virtual machine that can spin up in seconds. Within a few seconds? Yes, you heard me right or it can be done in milliseconds but depends on the OS you are using.
There are some blogs and articles that I found useful to get started with Firecracker like from Alex Ellis or Julia Evans or other articles that I found on the internet. They are discussing how to use Firecracker and build the image using Docker or without Docker in different ways.
During my exploration, I found that it’s not that easy to get started with Firecracker and understanding the concepts behind it. So, I decided to share what I’ve learned so far.
At the first, as always we are referring to the official documentation to get started with Firecracker that you can find here.
Basic pre-requisite is to have a Linux machine with KVM enabled or support for nested virtualization. If you are using cloud, you can use AWS or even GCP to create a VM with nested virtualization enabled.
You can check if your machine has KVM enabled by running the following command:
1
lsmod | grep kvm
If KVM is not enabled, you can enable it by running the following command:
1
modprobe kvm
Now, let’s download the Firecracker.
1
2
3
4
wget https://github.com/firecracker-microvm/firecracker/releases/download/v1.10.1/firecracker-v1.10.1-x86_64.tgz
tar -xvf firecracker-v1.10.1-x86_64.tgz
chmod +x release-v1.10.1-x86_64/firecracker-v1.10.1-x86_64
mv release-v1.10.1-x86_64/firecracker-v1.10.1-x86_64 /usr/local/bin/firecracker
Once done, the next step is to download the kernel and rootfs. Its required to run the microVM. Lets follow the steps from the official documentation to download it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ARCH="$(uname -m)"
latest=$(wget "http://spec.ccfc.min.s3.amazonaws.com/?prefix=firecracker-ci/v1.11/$ARCH/vmlinux-5.10&list-type=2" -O - 2>/dev/null | grep -oP "(?<=<Key>)(firecracker-ci/v1.11/$ARCH/vmlinux-5\.10\.[0-9]{1,3})(?=</Key>)")
# Download a linux kernel binary
wget "https://s3.amazonaws.com/spec.ccfc.min/${latest}"
# Download a rootfs
wget -O ubuntu-24.04.squashfs.upstream "https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/v1.11/${ARCH}/ubuntu-24.04.squashfs"
# Create an ssh key for the rootfs
unsquashfs ubuntu-24.04.squashfs.upstream
ssh-keygen -f id_rsa -N ""
cp -v id_rsa.pub squashfs-root/root/.ssh/authorized_keys
mv -v id_rsa ./ubuntu-24.04.id_rsa
# create ext4 filesystem image
sudo chown -R root:root squashfs-root
truncate -s 400M ubuntu-24.04.ext4
sudo mkfs.ext4 -d squashfs-root -F ubuntu-24.04.ext4
Reference: Getting a rootfs and Guest Kernel Image
The above commands will help us to get the rootfs and kernel image for Ubuntu 24.04 easily or if you want to build a custom image for your own use case, you can also use Docker to build the image. I’m not going to discuss that in this blog post, maybe in another part of the series.
At the beginning, I thought that Firecracker approach is similar to Docker that using an API to manage the microVMs. But, it’s not like that. Firecracker is a bit different from Docker. It’s not using a single API to manage the microVMs but own microVM has its own API. We will come to know more in the next experiment while starting the microVM.
Let’s continue by starting Firecracker process that running as a unix socket.
1
2
3
API_SOCKET="/tmp/firecracker.socket"
firecracker --api-sock "${API_SOCKET}"
After running the above command, you need to open another terminal to execute some commands to start the microVM.
Reference: Starting Firecracker
Once the Firecracker process is running, we can start the microVM by using the API but we need to create a network interface for the microVM using TAP device which can be done by using the following command:
1
2
3
4
5
6
7
8
9
10
TAP_DEV="tap0"
GATEWAY_IP="172.16.10.1"
FC_IP="172.16.10.100"
MASK_LONG="255.255.255.0"
ip tuntap add dev $TAP_DEV mode tap
ip addr add $GATEWAY_IP/24 dev $TAP_DEV
ip link set $TAP_DEV up
Do remember to enable IP forwarding in your machine.
1
sysctl -w net.ipv4.ip_forward=1
Otherwise, the microVM will not be able to access the internet using TAP device.
For better understanding about the TAP device, you can refer to this article a good article from Cloudflare.
TAP device is created, to make the guest VM able to access the internet, we need to add a route to the microVM using iptables.
1
2
3
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $TAP_DEV -o eth0 -j ACCEPT
Great! We are done with the network setup. Now, we can start the microVM by using the API. But hold on, how we can talk to the API if Firecracker is running as a unix socket?
We can use curl to talk to the API and use unix socket as the API endpoint.
Before starting the microVM, we need to create a configuration file for the microVM first.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Remember we are in different terminal hence need to set the API socket path again
API_SOCKET="/tmp/firecracker.socket"
KERNEL="./$(ls vmlinux* | tail -1)"
KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off ip=${FC_IP}::${GATEWAY_IP}:${MASK_LONG}::eth0:off"
ROOTFS="./ubuntu-24.04.ext4"
# Set the boot source for the firecracker microVM.
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{
\"kernel_image_path\": \"${KERNEL}\",
\"boot_args\": \"${KERNEL_BOOT_ARGS}\"
}" \
"http://localhost/boot-source"
Set the rootfs for the firecracker microVM.
1
2
3
4
5
6
7
8
9
10
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{
\"drive_id\": \"rootfs\",
\"path_on_host\": \"${ROOTFS}\",
\"is_root_device\": true,
\"is_read_only\": false
}" \
"http://localhost/drives/rootfs"
Set the network interface for the firecracker microVM.
1
2
3
4
5
6
7
8
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{
\"iface_id\": \"eth0\",
\"host_dev_name\": \"$TAP_DEV\"
}" \
"http://localhost/network-interfaces/eth0"
Note: The network interface name can be anything, it’s just a name and guest_mac is optional.
Next, set the machine configuration for the firecracker microVM such as the number of vCPUs and memory size because its not documented in the Getting Started guide. I come to know about it from OpenAPI specification file here.
1
2
3
4
5
6
7
8
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{
\"vcpu_count\": 2,
\"mem_size_mib\": 4096
}" \
"http://localhost/machine-config"
Last step is to start the firecracker microVM.
1
2
3
4
5
6
7
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{
\"action_type\": \"InstanceStart\"
}" \
"http://localhost/actions"
All the responses from the API requests we did above will be showing 204 No Content response which means the request is successful.
You can go back to the terminal where Firecracker process is running and it will start the microVM serial console. The question is how can we login to the microVM if we don’t know the username and password? Remember that we set the IP address in kernel boot arguments, so we can login to the microVM using SSH.
1
2
chmod 600 ubuntu-24.04.id_rsa
ssh -i ubuntu-24.04.id_rsa root@${FC_IP}
Unfortunately, the microVM will not be able to run apt update because we didn’t set the nameservers for the microVM.
1
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
Once we configure the nameservers, we can access run apt update and install some packages like regular Linux machine but inside the microVM!
Another way that you can do is to use configuration file to start the microVM without doing all the API requests manually like the previous steps. More information can be found here.
To kill the microVM, we can use the same request that we used to start the microVM but with SendCtrlAltDel action type.
1
2
3
4
curl -X PUT --unix-socket "${API_SOCKET}" -i \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
--data "{ \"action_type\": \"SendCtrlAltDel\" }" "http://localhost/actions"
For some reason, the above request is not working for me. I need to kill the firecracker process using kill command and remove its socket file. Or another way is to execute reboot command from the microVM itself after we SSH and the process will be killed.
After all the steps above, we can understand that Firecracker is using a different approach to manage the microVMs in simple way 1 socket per microVM. You might want to experiment to run 4000 microVMs on a single machine based on firecracker-demo project.
Other Resources: