TL;DR – Securely automate the decryption of your luks-encrypted Debian servers, to provide convenient yet substantial protection against data loss in the event of you losing some hardware. Here’s a video that goes through a real example of me encrypting a Debian 12 server using tang/clevis and showing the convenient yet substantially protective way it automatically decrypts my Debian servers on my home network:
The video clip includes me setting up two demo tang-servers in containers and using these servers to decrypt a luks-encrypted root file system on a Debian 12 server. All the instructions are in this blog post, but watching the video might make it easier for you to decide if this is for you. I encourage every Debian server user to luks-encrypt their drives to protect against data loss in the event of losing a disk (or even a whole server).
I ALWAYS luks-encrypt my linux drives, and I firmly believe you should too. Luks encryption is “full disk encryption”, and is typically used to protect against loss (including theft) of hardware. It ensures that your data are fully protected against physical hardware loss.
There are simply no exceptions to my rule of luks encrypting every drive I own. I do this because I don’t want my family’s data being accessed by anyone sleazy enough to e.g. break into my home and steal my hardware. It also makes it easy to dispose of a disk: disconnect if from power and put it in the trash. The contents are unreadable. No need to spend hours writing over each block of the disk to erase the contents. Quantum computing MAY make us revisit the impact of luks encryption, but today it’s still true: luks is solid.
Luks is a security measure and as such it can be “inconvenient” to use when it comes to boot disks and root file systems. To access luks-encypted drives you have to decrypt them, which typically means entering a password/key file to unlock them e.g. after power-up/reboot. For a desktop, that’s a completely acceptable task for me, but for bare-metal servers (and/or indeed virtual machine servers), this can be problematic.
There are TWO methods I use/have-used for decrypting luks servers: (1) dropbear-ssh (which allows you to gain ssh access to a luks-encrypted server during boot, so you can manually decrypt it) – this article is not about dropbear;
(2) A combination of a tang-server and clevis package to automate that process to a large degree (the subject of this article).
There are many good articles on how to use dropbear-ssh and it’s pretty easy to do (google is your friend for researching this further). There are also tutorials on how to use clevis/tang. Most of these seem to be for Red Hat or maybe Ubuntu servers. I struggled a little to get clevis to work on my Debian 12 (Bookworm) servers, so I wanted to document how I do it. This is because I want to make it easy for YOU to luks-encrypt your Debian servers and thus strongly protect from equipment-theft based threats.
This article is however a little more targeted in that it also intended for someone who has MORE THAN ONE physical server. If you only have one, I am not sure it’s really worth having a tang/clevis system, since your services go offline during a re-boot no matter what. I personally have THREE physical servers in my home-lab (because I am a little very crazy), and I work hard to make sure at least one of them is always up. So I configure clevis/tang to restart any two servers that get power-cycled. If all of my servers get power-cycled at once, they cannot automatically decrypt via clevis. I designed it that way deliberately.
So how does tang/clevis (two different software packages), work together to decrypt a luks header “automatically”? Well there are many excellent articles on that subject (e.g. here, and here, so please check them out), but in summary form, here’s what you do to use it, and a simplified version of how it works:
- Install Debian server via an iso installer onto your hardware (or virtual machine) as usual, but during the disk partitioning screen, select the option for ‘Guided, use entire disk and setup encrypted lvm’ during the partitioning screen, e.g. for the net-install method for Debian12:
- Enter a (very strong!!!) password at the ‘encryption passphrase’ prompt. This should use upper & lower case, symbols and numbers and should be awkwardly long to make brute-force impossible. For even better security, DO NOT USE A PASSWORD MANAGER, MAKE IT SOMETHING YOU CAN REMEMBER. NEVER WRITE IT DOWN, (Warning: if you forget this, bad things will happen to your data in terms of your ability to ever read them again!!).
- Complete the rest of the installation as normal, which ends with a reboot.
- Your first test: enter the luks encryption password when prompted to decrypt the OS root file-system and allow it to boot-up.
- Your server is now ready to have clevis installed and be configured to auto-decrypt your luks root file-system.
HOW DO I USE TANG/CLEVIS?
Before we start typing commands, here’s how I actually run my tang-clevis setup: I actually run TWO tang services, one of which is remote (not on my LAN, but still under my (remote) control), and the other does run on my LAN. In fact, the LAN-based tang-server is a high-availability cluster that can provide a single (identical) tang service to any machine on my LAN as long as at least one of the tang servers is online.
So my bare-metal servers have to get blinded decryption keys from two separate services for them to auto-decrypt at boot. One of which is provided by my highly-available tang server on the LAN, the other tang server is, well, somewhere else. This setup allows me to simultaneously reboot any TWO of my three LAN-based servers and they will automatically decrypt and reboot IF both tang services are available.
This is a deliberately fragile service: if anything is up with a tang service, chances are I have to manually decrypt my servers. In practice, I rarely do, so this just quietly runs and does its thing. I like that. Worse case, I can always vpn into my LAN, access a server via the IPMI interface and manually enter a luks code, but I am safe if e.g. anyone steals my hardware: any one of my physical servers that goes offline will NOT boot up if ANY of the following conditions are true:
- All servers have been power-cycled simultaneously (e.g. switched off/power-cut…or stolen!)
- Any server is disconnected from my local network prior to being rebooted (e.g. a thief back at their evil headquarters)
- If my remote tang-server goes offline for any reason (more on this below, because sometimes I make it go offline deliberately)
- And of course all the other reasons why services can fail…all of which are usually unintended of course!
I like all this, it allows me to sleep safe. I may lose my hardware, but I won’t ever lose our family data to a thief who steals my hardware (this method offers no protection against online hackers by the way – keep good offline backups!!).
For my remote tang-server, I am also experimenting with a systemd service that I have written, that does one extra thing: it tracks the status of my LAN servers to see if they are online. If they are, it keeps the remote tang-server hot and ready to serve blinded keys to them. If a LAN server goes offline, the systemd service sets a timer – enough time to allow for an automated reboot. If the server remains OFFLINE, for any reason, beyond my timer (e.g. in a scenario where someone has stolen my hardware, taken it somewhere and is attempting to power it up to see if they can somehow get my tang keys), the remote tang server is taken offline and the system it is on is rebooted (and thus deliberately locked via luks). No more remote tang-server until and unless I manually bring it back online. This extra complication essentially protects against even sophisticated thieves trying to steal a single server and use the online ones to somehow get to my data. If all goes well (and frankly it pretty much always does), my LAN servers automatically come back up after a reboot, but if any fragility kicks-in, I am there to do it manually. Very cool. As I said, I may lose my hardware to thieves, just not my data.
TECHNICAL DETAILS
We need to create a tang server (or better yet two or even more) to provide a tang service, and we need to install and configure clevis on each of the Debian servers we have luks-encrypted (in my case, that means all my bare metal servers). And then we need to link our tang services to a physical server’s boot luks header. Below, I will go through the simple steps needed to complete all of this.
Pre-requisite
I am assuming you have two or more Debian luks-encrypted servers that you want to configure clevis on. You need root access to these and details of IP address and basic networking (network gateway IP etc.).
I also assume you have a means of running a container on these servers, so we can install the tang services. These tang services have to be online, available and accessible to the luks-encrypted Debian server, so that at boot time the encrypted server can access these tang services on at least one of the other LAN tang servers in order to auto-decrypt the luks-encrypted root file system. In practice, this gives you a LOT of options (my own setup is a more complicated derivative of this), but I will try to keep it simple below.
Tang Server(s)
Tang is a service, like e.g. a web server. It just sits there listening at a port for a request to encrypt/decrypt a phrase. So it’s easy to setup. Just spin up your preferred method of virtualization (e.g. an incus or docker/lxd/lxc container, or a vm if you prefer). Ideally, this virtual instance should be installed on all of your encrypted servers so that it can provide a tang service on your LAN if any one of your servers is online. This is not essential, but I feel it’s a more secure way to implement tang/clevis.
Create and login to the soon-to-be tang server. You can use as many different servers as you like (I use two in this example) to create two or more tang servers. Per below, just enter and execute one command with root privileges:
sudo apt install tang joseHit enter, confirm the installation and let it complete. That’s all you need to do. Note down the IP address of the container and exit the container.
We can check that the tang service works by using a curl command. In the example below, I am assuming your IP address is 10.231.25.130. Change to suit:
curl http://10.231.25.130/adv
Just enter the above curl command (with the correct IP address) into a terminal and hit enter. Assuming you can connect to the server (i.e. on the same network) then the image below resembles what you should see (I use echo commands before and after, just to give some spacing in the output):
If you get a long string of similar looking “visual garbage” then…it’s working as it should! And that’s it. One tang server, ready to go. I recommend creating one on your LAN, and also one “somewhere else” (a distant friend/family’s computer, a hosting company that sells/rents vps’ etc. – somewhere pretty much out of the way). It’s not essential, it’s just how I do it. Once you set it up, just issue a ‘curl http:<ip-address>/adv’ command to fetch the advertising key per the above to see that it’s accessible as I show above. That’s it, one tang server ready to work for you (or indeed several of them).
[To make this even more secure, I recommend that you place at least one of these tang servers in a clustered high-availability mode on your network of encrypted physical servers. Doing so does mean that “at least one” of your LAN servers has to be online for this to work, which for most operations should be fine. If anyone wants to know more about this, just let me know (e.g. via @OGSelfHosting on X)]
Installing and Configuring Clevis
So now, let’s configure a luks encrypted server to coordinate with the tang services.
Login to your luks encrypted Debian server and enter a few commands as follows. Firstly, install clevis and it’s needed plugins:
sudo apt install clevis clevis-luks clevis-initramfsFor this next step we need networking information specific to your LAN. I am assuming you have TWO tang servers that you want to bind against a clevis key, setup as I showed earlier. I further assume these are accessible via the following ip addresses (change IP addresses to your tang servers):
http://tang-server1 (or, say, http://10.231.25.101)
http://tang-server2 (or, say, http://10.231.25.102)
I also assume:
- Your network has a gateway of 10.231.25.1/24, and thus has a netmask of 255.255.255.0;
- Your debian server’s luks root partition is named /dev/nvmeX-n1p3
- [You can use the lsblk command to find your /dev/ id]
- Your debian server’s network adapter is enp5s0
- [You can use ip a command to see your network adapters. If you have more than one, select the one that’s connecting to your router/gateway]
(Change these for your setup as you enter the commands below).
Enter this command in your debian server terminal, remembering to change the items in bold text for your setup:
sudo clevis luks bind -d /dev/nvmeX-n1p3 sss '{"t":2,"pins":{"tang":[{"url":"http://tang-server1"},{"url":"http://tang-server2"}]}}'When you use two servers, as I have done here, the clevis command uses a ‘shamir secret sharing’ (the ‘sss’ option), which basically instructs clevis to use two (or more) tang-servers to create a fragmented blinded luks key. If one of these tang-servers is on your LAN (especially on one of your encrypted servers), and one is in a distant location, this makes the service very very secure. Probably as good as it can be given that this is “automated”.
When you enter the above command, clevis asks for your luks decryption password (enter it accordingly), then it checks the tang servers and asks if you want to trust them (review and confirm). Clevis does some checks on the data and if there are no errors, it creates a new luks header key using a key derived from numbers sent back from the two tang-servers via a clevis-initiated prompt. Note that neither tang-server knows the luks password, they merely return a fragmented blinded code which on its own cannot decrypt a luks disk. Only clevis (and luks) know this password. Your old luks password WILL STILL WORK AS NORMAL but now there’s a new key that clevis can use to unlock the root partition at boot. But we are not done yet, we have to do some configuration so that there is network access during boot-up. Edit the following file on your Debian server:
sudo nano /etc/initramfs-tools/initramfs.confThe initramfs file allows you to specify information about your network. if you google initramfs you can find a lot more information, e.g. here. We are going to specify some of our networking – just enough to get an IP address on the LAN and internet access. Change these to suit your setup, then add this line at the end of the above file and save/quit. The items in bold will likely need changing. The network mask as shown below works for a /24 network (likely good enough for most people), but if you are running something more sophisticated then change this too.
IP=10.231.25.134::10.231.25.1:255.255.255.0::enp5s0
The IP config has to be right for clevis to work (but you can always edit this file after a boot, so it’s not world-ending if you make a mistake). Do not miss out the single and double colons. Google-search initramfs if you want to know more, but basically this sets ups most of the networking needed for clevis to go find a tang server. Save and quit the config file change when done. We have one more file to create, to allow us to potentially connect to remote tang servers (if required):
sudo nano /usr/share/initramfs-tools/hooks/curlPaste the following commands into the file (which I got from this link). It provides a name-server capability during boot then save/quit (so that tang can convert e.g. “remote-tang.mydomain.com” into an actual IP address), which you need if a tang-server is actually located on another network (i.e. remote):
#!/bin/sh -e
PREREQS=""
case $1 in
prereqs) echo "${PREREQS}"; exit 0;;
esac
. /usr/share/initramfs-tools/hook-functions
#copy curl binary
copy_exec /usr/bin/curl /bin
#fix DNS lib (needed for Debian 11)
cp -a /usr/lib/x86_64-linux-gnu/libnss_dns* $DESTDIR/usr/lib/x86_64-linux-gnu/
#fix DNS resolver (needed for Debian 11 + 12)
echo "nameserver 1.1.1.1\n" > ${DESTDIR}/etc/resolv.conf
#copy ca-certs for curl
mkdir -p $DESTDIR/usr/share
cp -ar /usr/share/ca-certificates $DESTDIR/usr/share/
cp -ar /etc/ssl $DESTDIR/etc/Now save the file/quit, make it executable then update the initramfs image. Then reboot as we are all DONE:
sudo chmod 755 /usr/share/initramfs-tools/hooks/curl
sudo update-initramfs -u -k 'all'
sudo reboot #This is optional, but it's how we test it. :)If everything works as expected, you will watch your server boot up, ask for a luks key, enable networking (get an IP, connect to the adapter etc.) then it will try and contact the tang servers. Assuming success, watch the service magically unlock your encrypted server drive and for the system to boot up as if you entered the luks key yourself. It’s a satisfying thing to watch!
You can stop a tang server to disable auto-luks decryption, and that is an extra layer of protection you can further configure to make this convenient yet substantially protective.
if you have avoided luks encryption because of the inconvenience of having to connect to your server every time you reboot, PLEASE think again. Tang and clevis make this super convenient yet I believe also extremely secure.
If there’s any interest, I’ll update on my ‘systemd’ optimization on my remote server, which I think makes this “better than a human” at protecting my luks-encrypted Debian servers during reboot, making it extremely particular about how a server can actually reboot and auto-decrypt.
I hope you found this tutorial useful. Message me if you have any questions or comments, e.g. at @OGSelfHosting on X.
Andrew