A distribution based on ZFSBootMenu to securely remote boot, rescue, and install Proxmox. This is an opinionated, but secure, and easy to use ZFSBootMenu-based distribution. (maybe the first?)
From curl to VM Playgroud in 60 seconds...
Partition, Encrypt and Install Proxmox...
First boot of Proxmox on encrypted partition, after install...
Let me try!
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/midzelis/zquickinit/main/zquickinit.sh)"
ZQuickInit
builds upon the foundations provided by ZFSBootMenu
with missing functions. Noteably, being able to access a ZFS-based box that exists in an untrusted environment behind a NAT with no UPNP or NAT-PMP services.
The box will use an encrypted root filesystem. The box itself may be stolen, so the encryption keys should not be stored in SecureBoot or TPM. In order to descrypt the drive, ZFSBootMenu
will prompt to load they key at startup. However, if this server has been restarted, it may be inconvient for the operator to enter the password.
This is solved by configuring the network interfaces and also adding Tailscale (SSH) to the ZFSBootMenu
configuration. Tailscale will allow remote access and will allow accessing the box behind the NAT. Tailscale also allows us to totally lock down the network side of things as well - there will be no listening ports on the local network interfaces at all - all of server will be running on the internal Tailscale interface.
However, ZFSBootMenu
chainloads the selected rootfilesystem after prompting for the decryption key. It boots the decrypted target kernel/initramfs - but this environment no longer has the original decryption key - and you'd have to enter the encryption key again.
Reprompting may just be merely inconvenient if you were in front of the box. But it is doubly inconvenient since this secondary kernel/initramfs environment would not have tailscale installed or even any network configuration at all. So this will stall out the remote boot process.
ZQuickInit
fixes this by 'injecting' the password into selected initramfs image. This takes advantage of the fact that Linux Kernel will continue to decompress initramfs images if it finds another initramfs image after the first one. So, to 'inject' the password, ZQuickInit
literally just creates a new compressed initramfs image and concatenates it with the selected initramfs image, and then boots that using the kexec.
Unfortunately, even with recent improvements to ZFSBootMenu
(the loadkey.d and botsel hooks), I wasn't able to convincingly implement this behavior. ZQuickInit
patches the entire load_key
and kexec_kernel
functions to implement this behavior.
ZFSBootMenu
is great because it is a fully featured (although significantly trimmed) linux distribution. However, I found it lacking for my purposes.
Instead of ending with secure decryption of the root filesystem - I saw potential as the basis of a full rescue utility, and installer for Proxmox to help bootstrap new machines.
ZQuickInit
can be started with no pools at all. In fact, you can drop the EFI on any FAT32 formatted USB stick and boot from it. You will be guided through a series of questions that will help you partition, create an (encrypted) pool, create a root dataset, configure a ESP partition (with ZQuickInit
installed on it) and finally a way to Install Proxmox into the new root dataset.
ZquickInit
can also be used to convert a previously unecrypted ZFS root partition into an encrypted one.
ZQuickInit
adds several useful programs that are useful when running a networked boot, specifally with disk/partitioning tasks in mind. This includes accessing LVM partitions, and ext3/4 and [ex]FAT[32] partitions.
To rescue or install, image a USB drive, or copy the EFI to a USB stick. Or, my favorite, use Ventoy to boot the EFI.
Since ZQuickInit
is designed to be run with SSH, and the console may also be active, a shared tmux
session is created between the local console and the remote SSH.
Getting tmux
running and also making the Installer and the various prompts pretty required a lot of trial and error to figure out keyboard mappings, unicode support, serial console, QEMU configuration, and so so many things. I'm proud to say that beatiful, colorful, unicode interfaces are supported however you access the ZQuickinit
- local console, SSH console, or via QEMU (vnc, or serial console).
ZFSBootMenu
has a lot of documentation. A lot. Its generally useful but slow to get started. It took me weeks to figure everything out. I hope that ZQuickInit
helps make ZFSBootMenu easier to try and use.
To support this goal, I make a lot of opinionated choices. Instead of supporting dracut and mkinitcpio, I choose to drop dracut. (I started with it at first, but it was hard to use, and slow, and cumbersome. Mkinitcpio is better in every single possible way - in my opinion!)
ZFSBootMenu
supports being built by almost any Linux distribution under the sun. I choose to drop all of these, and stick with just one: Void Linux. I never used this distro before, but I was pleasently surprised by it. Really, I chose it because it was the default distribution used by the Dockerfiles within ZFSBootMenu
. ZQuickInit
is exclusivesly build via Docker (or Podman, once they have support for syntax=docker/dockerfile:1
)
So I structured all of my "packages" that built on top of ZFSBootMenu
into a 'recipe' format. This is a simple convention-based package structure. First, it lives under /recipes
in the source tree.
It contains an optional recipe.yaml
metadata file. This file lists 2 things:
- the xbps dependencies
- help text
It contains an optional setup.sh
file. This file is invoked while the ZQuickInit
is being assembled/built. The main point of this file is to load additional data (configuration, secrets) into the image.
Then it creates a directory called initcpio
this folder contains all of the 'hooks' for mkinitcpio
- most importantly you'll want to use install
hook. The run
hook is used by ZFSBootMenu
itself, and probably shouldn't be used. Rather, you should use the existing hooks that ZFSBootMenu
exposes to add functionality.
You know how I said that the Linux Kernel will continue decompressing any cpio images found after the first one? Well, you can leverage this for ZQuickInit
itself!
That right, you don't even need to build anything to use ZQuickInit
- you can just download, and 'inject' your secrets of customizations directly into that image and then use it.
The main script is runnable directly from curl/bash. However, if you want more customization, or just hate this trend of random github repos asking you to blindly execute scripts from weird URLs, you can also just clone this repo and run it that way.
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/midzelis/zquickinit/main/zquickinit.sh)"
It will ask you if you want to download the EFI image, or run a "playground" that will boot the image in a QEMU VM.
git clone https://github.com/midzelis/zquickinit.git
cd zquickinit
./zquickinit.sh # this will show help screen
ZQuickInit image functions!
zquickinit.sh initramfs [--noask]
zquickinit.sh inject [target_efi] [source_efi]
zquickinit.sh iso [target_iso] [source_efi]
zquickinit.sh playground [--sshonly]
zquickinit.sh builder
zquickinit.sh tailscale
initramfs Build a zquickinit.efi image using docker or podman.
builder Advanced: Create the OCI builder image for ZQuickInit
tailscale Login to tailscale and save to tailscaled.conf
iso Build an iso image (for playground, or to write to USB)
playground Start a QEMU VM on the zquickinit.sh image in order to
play around with zquickinit.efi
--noask Do not ask any questions for building image.
Just use files in the current directory, if present.
--release Do not add QEMU debug
-d,--debug Advanced: Turn on tracing
-e,--enter Advanced: Do not build an image. Execute bash and
enter the builder image.
source_efi The zquickinit source image or download from GitHub image
if not specified.
target_efi Where the results of the image after injection will
be stored. Default is zquickinit.efi in the current folder
- First, you need to build the builder, which is the Void Linux based Docker image used to run mkinicpio.
./zquickinit.sh builder
- Then, you need to build ZQuickInit (this will be interactive), you can instead just use local files in the current directory instead, by passing
--noask
./zquickinit.sh initramfs
- When your ready to test, you can start the Playground in a QEMU VM. If you are going to be trying out the Proxmox installer (zquick_installer.sh) you can save bandwith by creating a folder
cache
in the top level source folder. This will be shared using 9p to the guest and will be used to cache downloaded apt packages.
./zquickinit.sh playground
- Secure
- Easy to build, customize
- Remote access to enter/load encryption keys
- Box must work behind NAT
- Support Proxmox as a first class environment
- Rescue/Install scenarios
- Speed
- it boots in less than <5 seconds (in QEMU at least)
- it doesn't need to be faster
- it boots in less than <5 seconds (in QEMU at least)
- Size
- its currently producing images around ~80MB
- could be smaller, but its 2023.
- its currently producing images around ~80MB
- Memory efficiency
- the initramfs is loaded into RAM at least twice
- but I don't care ;-)
- the initramfs is loaded into RAM at least twice
Important!
Anything you put in an initramfs image like ZFSBootMenu
will be stored unencrypted. What this means is that you should treat these keys as untrusted.
For Tailscale, you should create a new node dedicated to ZFSBootMenu
, and then configure ACLs to only allow incoming access to that node. Note: even tho this node won't be able to access the rest of your network, Tailscale will leak information about the status and tailnet IPs of the nodes on your network.
zquickinit.sh tailscale
Here's an example ACL:
{
"nodeAttrs": [
{
"target": ["tag:allowfunnel"],
"attr": ["funnel"],
},
],
"tagOwners": {
"tag:incomingonly": ["autogroup:admin"],
"tag:allaccess": ["autogroup:admin"],
"tag:allowfunnel": ["autogroup:admin"],
},
"acls": [
# only allow nodes tagged with allaccess to egress
{"action": "accept", "src": ["tag:allaccess"], "dst": ["*:*"]},
# to allow webttyd ingress
{"action": "accept", "src": ["*"], "dst": ["*:80"]}
],
"ssh": [
{
"action": "accept",
"src": ["tag:allaccess"],
"dst": ["tag:incomingonly"],
"users": ["autogroup:nonroot", "root"],
},
{
"action": "accept",
"src": ["tag:allaccess"],
"dst": ["tag:allaccess"],
"users": ["autogroup:nonroot", "root"],
},
],
}
Required dependencies, you can't remove this.
- Patches
zfsbootmenu-core.sh
- Prompts to unlock encrypted ZFS dataset.
- Then injects the passphrase into the boot environment to prevent prompting twice.
Defines Hooks:
/zquick/hooks/ifup.d
/zquick/hooks/ifdown.d
Uses Hooks:
/zquick/hooks/reboot.d
- Brings down interfaces
/libexec/hooks/early-setup.d
- Starts network interfaces/DHCP
- Installs
tailscale
configured as SSH server
Uses Hooks:
/zquick/hooks/ifup.d
- Starts tailscale
/zquick/hooks/ifdown.d
- Kills ssh sessions
Config Files:
tailscaled.state
->/var/lib/tailscale/tailscaled.state
tailscaled.conf
->/etc/tailscale/tailscaled.conf
- Installs
sshd
configured as a server running on a local interface
Uses Hooks:
/zquick/hooks/ifup.d
- Starts sshd
Config Files:
ssh_host_ed25519_key
->/etc/ssh/ssh_host_ed25519_key
ssh_host_ecdsa_key
->/etc/ssh/ssh_host_ecdsa_key
ssh_host_rsa_key
->/etc/ssh/ssh_host_rsa_key
authorized_keys
->/root/.ssh/authorized_keys
- Patches
zfsbootmenu-preinit.sh
to run the entirezfsbootmenu
script within atmux
session. - The tmux status bar will display status like hostname, tailnet name, and webttyd status.
Uses Hooks:
/zquick/hooks/bash.d
- Starting bash will automatically join the existing session (excluding nested sessions).
- Runs a web-based
ttyd
over tailscale funnel. - Wait, what? You serious??
- Its "secure-enough" in my opinion
- Generates a random 32 character URL
- Sets up
ttyd
to only respond to that URL - Send push notification to your phone using Pushover
- Notification has a link to random URL
ttyd
server will be killed after 30 minutes
Uses Hooks:
/zquick/hooks/ifup.d
/zquick/hooks/ifdown.d
Config Properties (in zquickinit.conf)
PUSHOVER_APP_TOKEN=aabbccddaabbccddaabbccdd PUSHOVER_USER_KEY=aabbccddaabbccddaabbccdd
- Provides script
zquick_installer.sh
- Partitioning
- Creating ZFS root pools/datasets
- Installs proxmox
- Encrypting existing uncrypted ZFS root pool/dataset
- Should only be used for testing. Not present in release image.
- Will mount
cache
and.
to/mnt/cache
and/mnt/qemu-host
over 9p filesystem
Defines Hooks:
/zquick/hooks/reboot.d
- Reboot command.
- Unmounts filesystems.
- Invokes hooks in
/zquick/hooks/reboot.d*
Sets hostname.
Config files:
hostname
->/etc/hostname
- vim
- nano
- ssh
- sshd
- curl
- wget
- nc
- w
- nmap
- ncat
- Sets font to ter-v16b, local console only.
Bunch of filesystem related utilities, including partitioning, efibootmgr, disk usage, etc.