Anatomy of a live Linux distribution

Introduction

Even though I didn’t find a formal definition, the terms “live Linux” or “live USB” (or “live CD/DVD”) are generally used to refer to an operating system not requiring an installation on the hard drive to run[1][2].

We already covered how to use some live distributions here (or touched the topic as a side note in other articles - follow the tag #live), but in this article we are going to show all the ingredients one by one. We will also apply the gathered knowledge to create a new Live system tailored to our needs.

Difference between a regular and a live Linux

From a technical perspective, one way to look at the differences between a regular installation of Linux and a live system is to further investigate the bootstrap process.

You might have heard that Linux is a monolithic kernel[4], which means it is supposed to be a single binary containing everything. Other architectures are possible, for example look up microkernels[5]. The monolithic approach favors simplicity, but the more subsystems and hardware support we integrate, the more increase in size we will have. Therefore, to avoid running out of memory just with the kernel, at compile time it is possible to disable unnecessary drivers and subsystems. In resource-constrained systems it is still required to compile the kernel, for servers, laptops or desktop computers it is not the case anymore. The distributions ship a binary kernel with enough features to bootstrap from most of the hardware currently in use.

Slackware Linux offers different flavors of the Linux kernel so the user can choose the best option depending on its hardware. This was very useful in the past, today the two options that are commonly used are huge and generic. The huge kernel tries to integrate in the same binary all the required drivers and in this case the bootstrap process is:

the bootloader loads in memory the file /boot/vmlinuz (the binary image of the kernel)
the kernel receives as a boot parameter the identifier of the main filesystem (for example root=/dev/sda1), a.k.a. root filesystem or simply rootfs
the kernel initializes itself, mounts the root filesystem and invokes the init process in it
the init process has PID=1 since it is the first process created by the operating system and is the root of the process tree.

In summary, the bootstrap procedure in the simplest form is:

bootloader -> kernel -> rootfs -> init

As a compromise between simplicity, optimization and flexibility they have introduced a way of splitting the kernel in the main binary vmlinuz plus a set of pluggable on-demand kernel modules. Note this is still a monolithic architecture because which modules are compiled is decided at compile time and when a module is loaded it becomes part of the kernel and runs in kernel space. In a micro-kernel arcitecture, for example, a device driver is just a regular process in user space and can be updated, compiled and restarted independently of the kernel.

We are mentioning kernel modules because there is an extra step in the most common situation:

bootloader -> load kernel and initramfs -> init (initramfs) -> rootfs -> init (rootfs)

The initramfs[6] (traditionally called initrd[7]) is a small filesystem with at least an init process and some kernel modules. The idea is to reduce vmlinuz to the essential features and compile everything else as a kernel module. Then we can create an initrd with only the modules required for our system. For example the root filesystem can be Ext4 or Xfs. If we build both drivers as modules, vmlinuz will not be able to mount the main filesystem directly but we only need one of these two depending on the filesystem chosen during installation. Let’s say we used Ext4, so in our initrd we will include the module ext4.ko. The bootloader will load the kernel and the initrd in the memory. Will leave the control to the kernel and pass the address of the initrd to the kernel. The kernel proceeds with the bootstrap. At some point it loads the module ext4.ko, mounts the main filesystem and continues eventually invoking init on the rootfs.

This procedure is more flexible and for example adapted very well when laptops started to have a solid-state drive (SSD) with interface NVMe[3]: it was only required to include the NVMe driver in the initramfs. It is also useful to support more elaborated situations where the root filesystem is managed by LVM, is encrypted or is on a read-only device like a CD-ROM. Last but not least, its power is leveraged to create live Linux distributions.

Before going into details let’s see from a high-level perspective a possible bootstrap sequence of a live Linux:

bootloader -> kernel + initramfs -> init (initramfs) create rootfs in RAM -> init (rootfs)

We are talking about possible bootstrap sequences because different options are available. For example many live systems use the syscall switch_root or pivot_root[10] to jump into the main filesystem, but Tiny Core Linux[8] just extends the initramfs and never leaves it[9]. We’ve already given other examples in the article on Slax and here is where you can find the complete init scripts of Porteus Kiosk and SalixOS Live.

Main technologies

We’ve done with the theory, let’s move to see each of the pieces we need to develop a functional proof of concept (POC).

Base distribution

We can pick our favorite Linux distribution as a base system. We are going to reuse all the userland. It doesn’t really matter which one you choose, so feel free to pick your favorite. In my POC I use Alpine Linux.

Kernel, Kernel Modules and Firmware

While the kernel is usually part of the base distribution, I had a problem with Alpine: the kernel works great in the Virtual Machine but it freezes when I try to run it on my laptop. So, if you are planning to use the kernel from your base distribution, feel free to skip this section and jump to Grub bootloader.

We’ve already seen how you can mix a kernel from a distribution with the rest of the software from another one in this article.

In my POC I have used the kernel from Slackware 15, but if you have Secure Boot enabled you might want to use Fedora’s one. Another good choice is to use the same kernel you are running right now (assuming you are on a Linux system).

Regardless of the source you need these files and folders:

boot/vmlinuz: the main image of the Linux kernel
lib/modules/: this folder contains the kernel modules (if you have multiple kernels installed on your machine, there is one folder for each kernel; the folder’s name is the kernel version)
lib/firmware/: this directory contains all the firmware

Useful commands to keep in mind if we are going to use Slackware’s components are:

# download the packages in the current folder
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-generic-5.15.19-x86_64-2.txz
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-modules-5.15.19-x86_64-2.txz
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-firmware-20220124_eb8ea1b-noarch-1.txz

# list the content of the packages
cat kernel-generic-* | tar Jt 
cat kernel-modules-* | tar Jt 
cat kernel-firmware-*| tar Jt 

# extract the folder boot from the package and install it in /
cat kernel-generic-* | tar Jx -C / boot
# extract the folder lib from the packages and install it in /
cat kernel-modules-* | tar Jx -C / lib
cat kernel-firmware-*| tar Jx -C / lib

In Slackware the packages are Tar archive compressed with XZ and all the firmware is packaged in a single archive.

If you prefer to use your running kernel you can simply get your kernel version with uname -r and copy the relevant files with:

cp /boot/vmlinuz $DEST_DIR/boot
KERNEL_VERSION=$(uname -r)
mkdir -p $DEST_DIR/lib/modules
cp -r /lib/modules/$KERNEL_VERSION $DEST_DIR/lib/modules
cp -r /lib/firmware $DEST_DIR/lib/

If you take the kernel from Fedora and you didn’t choose Fedora as a base distribution you might need a tool (like rpm2tgz) to eventually extract the content of the packages.

Grub bootloader

I am on a physical laptop and I have already Grub 2 installed. So, for my POC, I just need to add an entry in the file grub.cfg.

A basic Grub configuration is made of:

menuentry 'Live Linux' {
    # this line set the partition where Grub will load the kernel from
    set root=(hd0,gpt2)

    # load the kernel and give the partition of the root
    # filesystem
    linux   /boot/vmlinuz root=/dev/sda2

    # load the initramfs
    initrd  /boot/initrd.img
}

In the example above there are a couple of hardcoded values:

the root partition (hd0,gpt2)
the root filesystem /dev/sda2

A more robust approach to avoid the first point is to create a file on the root partition and then use it in the command search to set the root partition. For the second point it is better to use the filesystem label. In my POC I have hardcoded the root filesystem in the init script, so there is no need to pass it but for completion’s sake remember that most of the distributions support parameters like root=/dev/disks/by-label/yourlabel.

The Grub 2 configuration becomes:

menuentry 'Live Linux' {
    # search for the partition containing the file /myrootpartition
    # and set this partition as root (Grub will load the kernel from here)
    search -f /myrootpartition --set=root
    # load the kernel (the root parameter is commented because we will
    # hardcode the root filesystem in the init script of the initramfs
    linux   /boot/vmlinuz # root=/dev/disks/by-label/mylabel

    # load the initramfs
    initrd  /boot/initrd.img
}

If you are using a Virtual Machine (like on VirtualBox) you are probably in my situation, however if you need to install Grub on a pen drive (you are creating a live USB) refer to this article or, if you want a dual boot system you can install grub following Step 4 here.

chroot

To be able to create the main filesystem of our live distribution we need to install the base distribution in a folder. We call this folder chroot environment because we can jump into the folder with the command chroot and processes perceiving this folder as the origin of the virtual filesystem. In other words, if this folder was a partition on the hard drive we could use it as the root filesystem.

This is an operation we need to perform also to create docker images from scratch. Some distributions have easier tools than others, but we can always achieve this kind of installation. For Debian (and derivates) there is the command debootstrap, in Fedora we used dnf -installroot= in this article and for Slackware we described the full procedure here.

In our POC we are going to use Alpine Linux and the installation in chroot is described very well in its wiki[11]. We have already used this procedure in another article in this section. Basically it requires downloading a statically linked version of the Alpine package manager APK and running it with few parameters. Useful commands here are:


wget https://dl-cdn.alpinelinux.org/alpine/v3.21/main/x86_64/apk-tools-static-2.14.6-r3.apk
cat apk-tools-static-2.14.6-r3.apk | tar zx sbin/apk.static
mv sbin/apk.static .

./apk.static -X https://dl-cdn.alpinelinux.org/alpine/v3.21/main \
    -U --allow-untrusted -p ./${chroot_dir} --initdb \
    add alpine-base

Once the base chroot folder is set up, we can proceed with installing more packages and configuring the system inside the chroot environment. As discussed also here we can use the feature mount --bind to mount the same content of the file /etc/resolv.conf (necessary for the DNS resolution in the chroot) and the folders /dev, /sys, /proc inside the chroot environment:

# the variable CHROOTDIR point to the folder where we installed Alpine

for i in etc/resolv.conf dev proc sys; do
    mount --bind $i $CHROOTDIR/$i
done

chroot $CHROOTDIR /bin/bash

If we want to execute a script instead of having an interactive shell we do for example:

cp myscript.sh $CHROOTDIR/tmp
chroot $CHROOTDIR /tmp/myscript.sh

SquashFS

We have now a full operating system installed in a folder, we can chroot into it to perform installations, but how do we use this as a root filesystem in our live Linux?

In theory we can just create a Tar archive and extract the content during the bootstrap process, but what if we don’t have enough RAM to hold all the extracted content? We can create a compressed filesystem image with SquashFS and then copy the compressed image to RAM and mount it. Once it is mounted we have access to all the content, but the driver is capable of accessing the desired data without decompressing the whole archive.

To create the SquashFS image we can run:

cd $CHROOTDIR
tar c * | sqfstar -comp xz -b 1M $OUTDIR/myfilesystemimage.sqfs

To mount the image we need the kernel modules loop and squashfs.

# load the kernel modules in case they are not already loaded
modprobe loop 
modprobe squashfs

# check what is the first available loop device block
losetup -a

# associate /dev/loop0 to $OUTDIR/myfilesystemimage.sqfs
losetup -f /dev/loop0 $OUTDIR/myfilesystemimage.sqfs

# mount the filesystem
mount -t squashfs -oro /dev/loop0 /mnt

tmpfs

One issue with SquashFS is that it is read-only, but we need a writable filesystem if we want to use it as a root filesystem. We will fully solve this issue in the next section, but a new ingredient need to be added here: tmpfs.

The idea behind tmpfs is to have a volatile filesystem that lives completely in RAM. So it is possible to create files and directories, special files and everything but all the content will be lost on reboot. This is ideal for example for temporary files or files to be cached. Mounting a tmpfs is as simple as running mount -t tmpfs mynewfs /mnt. The word mynewfs is an arbitrary identifier of the filesystem shown by df -h.

overlayfs

So how do we use the power of SquashFS and tmpfs to create a writable filesystem for our new live Linux?

We can use overlayfs[12] to merge two different filesystems. This is not the only solution (for example Slax uses AuFS, others use UnionFS), but this is supported natively by the kernel.

To mount the overlayfs we need:

a lower directory
an upper directory
an empty working directory

OverlayFS merges the content of the lower directory with the upper directory. The working directory must live on the same filesystem as the upper one. It is possible to have all folders on the same filesystem and there is no need for any of them to be the root of a filesystem. To support a writable overlayfs, the filesystem of the upper tree must support specific features so, for example, NFS cannot be used. In the opposite case (read-only overlay) any filesystem type can be used.

Multiple overlayfs can be merged into other overlayfs. In most of the live distributions this feature is used to divide a big filesystem into modules of smaller size. On the other side it increases the complexity of the system, so we are not going to take advantage of this in our experiment.

To have a writable filesystem suitable to our case we need the following directories:

/media/lowerdir: we will mount squashfs read-only here
/media/root-rw: mount point of tmpfs (writable)
/media/root-rw/upperdir: upper directory
/media/root-rw/workdir: working directory
/sysroot: mount point of overlayfs

In /sysroot we will have the merged filesystem (suitable to be the new root filesystem). /media/root-rw/upperdir is an empty folder so the content of /sysroot will be equal to /media/lowerdir. If we create/delete/edit a file or a folder in /sysroot the working directory will store all the changes.

SRCFILE=myfilesystemimage.sqfs
LOWERDIR=/media/lowerdir
TMPFSDIR=/media/root-rw
UPPERDIR=$TMPFSDIR/upperdir
WORKDIR=$TMPFSDIR/workdir
sysroot=/sysroot

# Mount read-only underlying rootfs
mkdir -p $LOWERDIR
mount -t squashfs -o ro $SRCFILE $LOWERDIR

# mount writable filesystem: tmpfs
mkdir -p $TMPFSDIR
mount -t tmpfs -o mode=0755,rw root-tmpfs $TMPFSDIR
# create upperdir and workdir on the same filesystem
mkdir -p $UPPERDIR $WORKDIR

# Mount writable overlay on /sysroot
modprobe overlay
mkdir -p $sysroot
mount -t overlay -o \
	lowerdir=$LOWERDIR,upperdir=$UPPERDIR,workdir=$WORKDIR \
	overlayfs $sysroot

initramfs

The last ingredient of our live Linux is the initramfs. The essence is a cpio compressed archive. As shown in Pocket Linux Guide[13] the filesystem can be as simple as the binary (statically linked) of a shell and a device block for the console. In practice we need to populate with all the commands we need, their dependencies, kernel modules, etc.

Looks like a lot of work, but luckily most distributions provide a shell script to facilitate this operation. Run the command after the installation of the kernel, the kernel modules and the firmware. It doesn’t matter if you took those files from a different distribution, the script will work (use the script provided by your base distribution). Run it in the chroot environment.

After that we are going to extract all the content, replace or customize the init script and then recreate the cpio archive.

To create the file initrd check the documentation of the base distribution of your choice. For example:

# ===> Fedora Linux
dracut --force --no-hostonly /boot/initrd.img ${KERNEL_VERSION}`

# ===> Slackware Linux
mkinitrd -c \
    -k \$(/bin/ls /lib/modules) \
    -m nvme:ext4:exfat:vfat:squashfs:loop \
    -r /dev/disk/by-label/fed2 \
    -f ext4

# ===> Alpine Linux
mkinitfs -c /etc/mkinitfs/mkinitfs.conf -b / $(/bin/ls /lib/modules)

In our POC we are using Alpine Linux, so after the installation of kernel/modules/firmware from Slackware, we can install the package mkinitfs with APK and run the command to generate the file /boot/initramfs-vanilla in the chroot environment.

Let’s extract the content, read the init script and then, if it is not too complicated, let’s reuse this one as a base to create ours.

mkdir extr
cat initramfs-vanilla \
    | gzip -c -d \
    | cpio --directory=extr --extract
mv extr/init extr/init_orig

the command cpio --directory=extr --extract using the synthetic version of the options is cpio -D extr -i.

Open the file extr/init_orig in a text editor and see if it is comprehensible enough to be used as a base for our custom script. The code looks simple enough and since we are just developing a POC we will reuse the code as much as possible.

Let’s create also an init script that just executes a shell. Nothing more.

cat <<"EOF" > extr/init_shell
#!/bin/sh

echo "=== init_shell ==="
echo "This init script just run a shell. No other initialization is performed"
echo "Use '/bin/busybox --install -s' to create regular symbolic links"
echo "'export PATH="$PATH:/usr/bin:/bin:/usr/sbin:/sbin' if necessary"
exec /bin/busybox sh
reboot
EOF
chmod 755 extr/init*

Notice the technique of running reboot after the exec command so, in case of failure, the system simply reboots instead of going into kernel panic (I have seen this in the original script).

In the next section we will put all the concepts together and we will show the custom init script. We will end up with these three files:

init: our custom init script
init_shell: an init script that just runs a shell
init_orig: the original init script

There is a boot parameter we can give to the kernel from Grub to tell the kernel to execute another program as init script. For example we can pass rdinit=/init_shell to have the second script invoked and perform some experiment or debug inside the initramfs during the bootstrap. Note there is also the parameter init= but this one affects which program is invoked as init in the root filesystem[14].

Once the initrd image is customized we can recreate the archive with

pushd extr
find * | cpio -o -H newc | gzip -c > ../initrd
popd

Alpine uses gzip for compression, but other distributions might prefer xz.

Create a new live Linux

Let’s put everything together and develop a working live system. I will install outside of the chroot only the base system, but then I will run chroot and proceed inside with this script (saved with name bsdrulez_live__firstconfig.sh:

#!/bin/sh

# bsdrulez_live__firstconfig.sh

### ACTIVATE REQUIRED SERVICES AT STARTUP ###
# from https://wiki.alpinelinux.org/wiki/Alpine_Linux_in_a_chroot#Preparing_init_services
rc-update add devfs sysinit
rc-update add dmesg sysinit
rc-update add mdev sysinit

rc-update add hwclock boot
rc-update add modules boot
rc-update add sysctl boot
rc-update add hostname boot
rc-update add bootmisc boot
rc-update add syslog boot

rc-update add mount-ro shutdown
rc-update add killprocs shutdown
#rc-update add savecache shutdown

# NOTES:
#
# linux-lts depends on linux-firmware-any
# and it installs 600MB of firmware.
# Use linux-firmware-none, or install only the
# desired (e.g. linux-firmware-mediatek, linux-firmware-radeon)
#
# Alpine kernel (linux-lts) does not boot on my laptop, but
# I was able to successfully use Slackware kernel/modules/firmware
#
# IMPROVEMENT: 1. firmware is ~700MB, trim down only the required
#              maybe using the Alpine packages
#              2. run setup-xorg-base
#              3. apk search xf86- | grep -v -e -dev- -e -doc-
#              4. apk add openbox rxvt-unicode alsa-utils
#              5. cutomize /etc/asound.conf for card 1 as default
#              6. apk add openssh + add keypairs
#              7. add net and batt scripts

### INSTALL KERNEL, MODULES AND FIRMWARE FROM SLAKWARE ###
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-generic-5.15.19-x86_64-2.txz
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-modules-5.15.19-x86_64-2.txz
wget https://slackware.uk/slackware/slackware64/slackware64/a/kernel-firmware-20220124_eb8ea1b-noarch-1.txz
cat kernel-generic-* | tar Jx -C / boot
cat kernel-modules-* | tar Jx -C / lib
cat kernel-firmware-*| tar Jx -C / lib
rm kernel-*

### INSTALL OTHER SOFTWARE AND GENERATE INITRAMFS ###
cat <<"EOF" | xargs apk add
mkinitfs
wpa_supplicant
openssl
gpg
EOF

# IMPROVEMENT: might be useful to add exfat (fat already included)
#              check /etc/mkinitfs/features.d for more details
cat <<"EOF" > /etc/mkinitfs/mkinitfs.conf
features="ata base cdrom ext4 squashfs keymap kms mmc nvme raid scsi usb virtio"
EOF

# tried option -i /root/init, but it doesn't work
mkinitfs -c /etc/mkinitfs/mkinitfs.conf -b / $(/bin/ls /lib/modules)

As you can see the script does some configurations for the services that need to be started during the bootstrap, it downloads and installs the kernel, the kernel modules and the firmware. Then it proceeds with installing some packages (I want wpa_supplicant to be able to connect to the Wifi network), configuring and running the script mkinitfs to create the initramfs.

Let’s proceed with our custom init script and create the file myinit.sh:

#!/bin/sh

# myinit.sh

echo "=== CUSTOM INIT SCRIPT IN INITRAMFS ==="

### CONSTANTS ###
# this is the init script version
VERSION=3.11.1-r0
sysroot="$ROOT"/sysroot

### FUNCTIONS ###
ebegin() {
	last_emsg="$*"
	echo "$last_emsg..." > "$ROOT"/dev/kmsg
	[ "$KOPT_quiet" = yes ] && return 0
	echo -n " * $last_emsg: "
}
eend() {
	local msg
	if [ "$1" = 0 ] || [ $# -lt 1 ] ; then
		echo "$last_emsg: ok." > "$ROOT"/dev/kmsg
		[ "$KOPT_quiet" = yes ] && return 0
		echo "ok."
	else
		shift
		echo "$last_emsg: failed. $*" > "$ROOT"/dev/kmsg
		if [ "$KOPT_quiet" = "yes" ]; then
			echo -n "$last_emsg "
		fi
		echo "failed. $*"
		echo "initramfs emergency recovery shell launched. Type 'exit' to continue boot"
		/bin/busybox sh
	fi
}

### MAIN SCRIPT ###
/bin/busybox mkdir -p \
	"$ROOT"/bin \
	"$ROOT"/sbin \
	"$ROOT"/usr/bin \
	"$ROOT"/usr/sbin \
	"$ROOT"/proc \
	"$ROOT"/sys \
	"$ROOT"/dev \
	"$sysroot" \
	"$ROOT"/media/cdrom \
	"$ROOT"/media/usb \
	"$ROOT"/tmp \
	"$ROOT"/etc \
	"$ROOT"/run/cryptsetup

# Spread out busybox symlinks and make them available without full path
/bin/busybox --install -s
export PATH="$PATH:/usr/bin:/bin:/usr/sbin:/sbin"

# Make sure /dev/null is a device node. If /dev/null does not exist yet, the command
# mounting the devtmpfs will create it implicitly as an file with the "2>" redirection.
# The -c check is required to deal with initramfs with pre-seeded device nodes without
# error message.
[ -c /dev/null ] || $MOCK mknod -m 666 /dev/null c 1 3

$MOCK mount -t sysfs -o noexec,nosuid,nodev sysfs /sys
$MOCK mount -t devtmpfs -o exec,nosuid,mode=0755,size=2M devtmpfs /dev 2>/dev/null \
	|| $MOCK mount -t tmpfs -o exec,nosuid,mode=0755,size=2M tmpfs /dev

# Make sure /dev/kmsg is a device node. Writing to /dev/kmsg allows the use of the
# earlyprintk kernel option to monitor early init progress. As above, the -c check
# prevents an error if the device node has already been seeded.
[ -c /dev/kmsg ] || $MOCK mknod -m 660 /dev/kmsg c 1 11

$MOCK mount -t proc -o noexec,nosuid,nodev proc /proc
# pty device nodes (later system will need it)
[ -c /dev/ptmx ] || $MOCK mknod -m 666 /dev/ptmx c 5 2
[ -d /dev/pts ] || $MOCK mkdir -m 755 /dev/pts
$MOCK mount -t devpts -o gid=5,mode=0620,noexec,nosuid devpts /dev/pts

# shared memory area (later system will need it)
mkdir -p "$ROOT"/dev/shm
$MOCK mount -t tmpfs -o nodev,nosuid,noexec shm /dev/shm


# read the kernel options. we need surve things like:
#  acpi_osi="!Windows 2006" xen-pciback.hide=(01:00.0)
###BRLZ set -- $(cat "$ROOT"/proc/cmdline)


echo "Alpine Init $VERSION" > "$ROOT"/dev/kmsg
echo "Slackware Kernel generic 5.15.19" > "$ROOT"/dev/kmsg

if grep -q -F -w rescue /proc/cmdline; then
    exec /bin/busybox sh
    reboot
fi

# load available drivers to get access to modloop media
ebegin "Loading boot drivers"

$MOCK modprobe -a loop squashfs 2> /dev/null
if [ -f "$ROOT"/etc/modules ] ; then
	sed 's/\#.*//g' < /etc/modules |
	while read module args; do
		$MOCK modprobe -q $module $args
	done
fi
eend 0

# zpool reports /dev/zfs missing if it can't read /etc/mtab
ln -s /proc/mounts "$ROOT"/etc/mtab


# mount SRCDEV to SRCFOLDER to get the squashfs file
# TODO: remove this hardcoded stuff - use boot params
SRCDEV=/dev/nvme0n1p6
SRCPATH=alpinelive/alpinelive.sqfs
SRCFILE=$(echo $SRCPATH | rev | cut -d / -f 1 | rev)
SRCFOLDER=/media/srcfolder
LOWERDIR=/media/lowerdir
TMPFSDIR=/media/root-rw
UPPERDIR=$TMPFSDIR/upperdir
WORKDIR=$TMPFSDIR/workdir
mkdir -p $SRCFOLDER \
         $LOWERDIR \
         $TMPFSDIR \
         $sysroot

# use blkid to figure out filesystem type
mount -t ext4 -o ro $SRCDEV $SRCFOLDER
cp $SRCFOLDER/$SRCPATH /
umount $SRCFOLDER

# Mount read-only underlying rootfs
mount -t squashfs -o ro /$SRCFILE $LOWERDIR

# Mount writable filesystem (upper and workdir must be on the same fs)
overlaytmpfsflags="mode=0755,rw"
$MOCK mount -t tmpfs -o $overlaytmpfsflags root-tmpfs $TMPFSDIR
mkdir -p $UPPERDIR $WORKDIR

# Mount writable overlay
modprobe overlay
$MOCK mount -t overlay -o \
	lowerdir=$LOWERDIR,upperdir=$UPPERDIR,workdir=$WORKDIR \
	overlayfs $sysroot


cat "$ROOT"/proc/mounts 2>/dev/null | while read DEV DIR TYPE OPTS ; do
	if [ "$DIR" != "/" -a "$DIR" != "$sysroot" -a -d "$DIR" ]; then
		mkdir -p $sysroot/$DIR
		$MOCK mount -o move $DIR $sysroot/$DIR
	fi
done

$MOCK sync

exec switch_root  $sysroot /sbin/init

echo "initramfs emergency recovery shell launched"
exec /bin/busybox sh
reboot

The first part of the script is exactly what you can find in the original init script except for parts that we have removed because we were not using them. As you can see in the code, the script installs the symbolic links of Busybox, mounts some support filesystems and loads kernel modules (we have probably added squashfs). From the TODO comment downward we have developed our custom code and we have reused a few snippets. In the original code it is annoying the use of empty variables $ROOT and $MOCK, but I hadn’t time to clean the code yet. We can use those to see what I have reused from the original init.

Since Alpine uses mdev we need to update the list of modules we want the system to load during the bootstrap. To do that we can peek what are the current modules running and prepare a list extracting the first column from the second line (since the first is a header):

lsmod | awk '{print $1}' | tail -n +2 > modules

We are now ready to develop the main script that uses the 3 files just created. Call this script bsdrulez_live__create_alpine.sh.

#!/bin/bash

# bsdrulez_live__create_alpine.sh

[[ -n "$1" ]] && MANUALFIRSTCONFIG=true

DSK=alpine-core.dsk
LBL=mylive
SCRIPTDIR=$(dirname $(realpath $0))
FIRSTCONFIG=$SCRIPTDIR/bsdrulez_live__firstconfig.sh
WORKDIR=$(realpath $PWD)/workdir
ROOTFS=$WORKDIR/rootfs
OUTDIR=/dev/shm/alpinelive

# there is a symbolic link in 'latest-stable' that point
# to 'v3.21' at the time of writing, but the version of
# apk-tools-static is hardcoded in APK_STATIC_URL
MIRROR=https://dl-cdn.alpinelinux.org
APKREPO=${MIRROR}/v3.21/main
APK_STATIC_URL=${APKREPO}/x86_64/apk-tools-static-2.14.6-r3.apk


# create disk file and mount in a folder
fallocate -l 2G $DSK
mkfs.ext4 -O ^has_journal -m 0 -L $LBL $DSK
mkdir $WORKDIR
mount -t ext4  $DSK $WORKDIR
cd $WORKDIR

# create rootfs and install base system
mkdir $ROOTFS
curl -LO $APK_STATIC_URL
tar -xzf apk-tools-static-*.apk
./sbin/apk.static -X ${APKREPO} \
    -U --allow-untrusted \
    -p ${ROOTFS} \
    --initdb add \
    alpine-base
# alpine-base include the bare minumum: APK, Busybox, openrc - 11 MB

# prepare chroot
mount -o bind /dev ${ROOTFS}/dev
mount -t proc none ${ROOTFS}/proc
mount -o bind /sys ${ROOTFS}/sys
echo 'nameserver 8.8.8.8' > $ROOTFS/etc/resolv.conf
chattr +i $ROOTFS/etc/resolv.conf
mkdir -p ${ROOTFS}/etc/apk
echo "$APKREPO" > ${ROOTFS}/etc/apk/repositories

# install relevant software and perform configuration
if [ -n "$MANUALFIRSTCONFIG" ]; then
    chroot ${ROOTFS} /bin/sh -l
else
    cp $FIRSTCONFIG $ROOTFS/root/firstconfig.sh
    cp $SCRIPTDIR/modules $ROOTFS/etc/modules
    chroot ${ROOTFS} /bin/sh -x /root/firstconfig.sh
fi
umount ${ROOTFS}/{dev,proc,sys}

# generate required files to boot
pushd $ROOTFS/boot 
cat <<"EOF" > grub.cfg
menuentry 'BSDrulez Live - Alpine userland with Slackware kernel' {
        #set root=(hd0,gpt1)
        search -f /alpinelive/vmlinuz -s root
        linux   /alpinelive/vmlinuz ipv6.disable=1  # rdinit=init_shell   rescue
        initrd  /alpinelive/initrd
}
EOF

mkdir extr
cat initramfs-vanilla \
    | gzip -c -d \
    | cpio --directory=extr --extract
mv extr/init extr/init_orig
cp $SCRIPTDIR/myinit.sh extr/init

cat <<"EOF" > extr/init_shell
#!/bin/sh

echo "=== init_shell ==="
echo "This init script just run a shell. No other initialization is performed"
echo "Use '/bin/busybox --install -s' to create regular symbolic links"
echo "'export PATH="$PATH:/usr/bin:/bin:/usr/sbin:/sbin' if necessary"
exec /bin/busybox sh
reboot
EOF
chmod 755 extr/init*

pushd extr
find * | cpio -o -H newc | gzip -c > ../initrd
popd
rm -rf extr
popd # return from $ROOTFS/boot

mkdir $OUTDIR
cp $ROOTFS/boot/{vmlinuz*,initrd,grub.cfg} $OUTDIR
echo "Kernel, initrd and grub.cfg can be found in $OUTDIR"

# create squash image
pushd $ROOTFS
tar c * | sqfstar -comp xz -b 1M $OUTDIR/alpinelive.sqfs
echo "Created $OUTDIR/alpinelive.sqfs"
popd

After the explanation in the previous section the script should be straightforward. The only additional step is the creation of a big file with fallocate that I format with a filesystem and mount using a loop device to have a working directory in /dev/shm with execute permission (on my system the tmpfs mounted under /dev/shm doesn’t allow command execution). Feel free to remove these lines since they are not really part of the POC. Remind also the poor code quality it’s because we just want to see if our concepts are enough to create a live Linux that runs entirely from RAM. You will need to substitute at least all the hardcoded paths if you want to run it on your machine (or in a Virtual Machine).

Running the POC

To run the POC create an empty folder and populate it with the following files from the previous section:

bsdrulez_live__firstconfig.sh
bsdrulez_live__create_alpine.sh
modules
myinit.sh

The script will create the root filesystem in a SquashFS image. During the bootstrap of the live system, this image needs to be found from the initramfs so plan a partition that will hold the image and take note of its device block. Put this identifier instead of /dev/nvme0n1p6 at line 114 of myinit.sh. If the filesystem is not Ext4, change also line 128. It is important to use the real device block as opposed to other identifiers (like for example /dev/disks/by-label/mylabel) because the initrd of Alpine does not support it (if you are using another base distribution it will probably work).

Run ./bsdrulez_live__create_alpine.sh and then check the folder /dev/shm/alpinelive. The script creates this folder and place all the relevant artifacts there. You should have the kernel (vmlinuz), the initramfs (initrd), an example of entry to put in the main grub.cfg and the main filesystem alpinelive.sqfs.

Now mount the partition identified at line 114 of myinit.sh (mine is /dev/nvme0n1p6, but you have just changed it two paragraphs ago) if it is not already mounted and copy the folder alpinelive directly under the mount point. For example in my case I did:

# replace the value of SRCDEV with the device block of your partition
SRCDEV=/dev/nvme0n1p6

mount $SRCDEV /mnt
cp -r /dev/shm/alpinelive /mnt
ls /mnt/alpinelive/alpinelive.sqfs  # this is just a check

Rename the kernel image in alpinelive to vmlinuz in case it has a different name. Copy the new bootloader entry from alpinelive/grub.cfg to the relevant section in the file grub.cfg of your local installation of Grub.

Reboot and select the new entry. You should be able to log in to the live system with root (no password). Run df -h to verify there is no mount point from any persistent device.

Proceed now configuring the network:

# check what are the name of the available interfaces
ifconfig -a

# WPA configuration (only in case you want to connect to a Wifi network
NETWORK=YourWifiNetworkName
PASSWORD=YourWifiPassword
wpa_supplicant -D nl80211 -i wlan0 -c  <(wpa_passphrase $NETWORK $PASSWORD) -B

# Run the DHCP client to get an IP address (use eth0 if it's ethernet)
udhcpc -i wlan0

Test the connection by installing vim, curl and wget:

apk update
apk add vim curl wget

If everything works fine you can proceed with the installation of a full desktop environment and the creation of a regular user:

# There is also a script "setup-desktop" if you prefer
apk add xfce4 xfce4-terminal xfce4-screensaver font-dejavu adwaita-xfce-icon-theme adw-gtk3

adduser user
addgroup user input
addgroup user video
addgroup user audio

Move to a different terminal using Alt+Right or Ctrl+Alt+F4 and log in as user, Create the file ~/.xinitrc and start the desktop session:

echo 'exec startxfce4' > ~/.xinitrc
startx

The new desktop environment should start without any issues. Install the browser and play a video to test that everything is working fine:

# run this command in a xfce4-terminal
su - -c 'apk add alsa-utils pulseaudio pulseaudio-alsa pavucontrol firefox chromium'
pulseaudio --start
pavucontrol &
firefox &

In my case I had to set the default audio card, but probably yours will work out of the box.

Troubleshooting and hijacking init

If your system doesn’t boot properly, you can pass to the kernel the boot parameter rescue (at the Grub prompt or in grub.cfg) and our custom init script in the initrd, will drop in an interactive shell.

If it doesn’t reach that point you can try with rdinit=/init_shell. This will interrupt the boot process as soon as you enter the initramfs.

I was excited to have this new system working, but since it is a live Linux all the work will be lost on every reboot. To test new configurations we can customize the content of bsdrulez_live__create_alpine.sh, but the process of recreating the SquashFS is long enough to slow down the development significantly.

For this reason you can move a part of the configuration in the initrd (and use the init script to install the new configuration). This is quicker for sure, but still it is not convenient to unpack and repack the initrd every time you want to try a new configuration.

For this reason the best solution is to create a script in the folder alpinelive and add a line in the init script so this new script gets invoked during bootstrap. When you want to change the operations performed you simply update the content of this script (recreating the initrd all the time is not required anymore). I called this solution “init hijacking” because I pushed this approach to the extreme and have externalized all the second part of the init script as well,

Eventually I will be happy with all of my tests and I will come up with a configuration I am happy with and will recreate the SquashFS.

Further development

I was happy to have this project done and see it working. Except for having fun and sharpening my knowledge, this POC can actually be used for several interesting projects:

Finalize a configuration with all the tools and configuration I need so I can use it as a regular OS where most of the time the hard drive is unmounted. This is useful for example if you are concerned about the limited write cycle of your solid-state drive or if your main is to browse the web and you keep deleting cache and cookies after every session.
Generalize the script using a specific setup script for the chroot to support userland from different distributions
create my own live Linux distribution where I create a set of prebuilt images with the most common configurations
create a simple way for interested people to try Linux on a laptop without installing it

Conclusions

In this article we explained the theory behind a live Linux distribution, we moved to the main concepts with a hands-on understanding of all the tools involved and eventually we brought everything together and created a full live system.