TL;DR – here’s a video that explains briefly what cockpit is, and then goes though how it can be installed, and more importantly how 2FA can be enabled to make access more secure than the out-of-the-box default of username/password. I also go over where to find other plugin applications that can help make Cockpit even more useful. Enjoy!
[Update – video now in MP4 format – thanks to @[email protected] for pointing this out]
Cockpit is an open source project sponsored by Red Hat to provide a simple GUI management portal that aids linux server management. Installation is a breeze (‘sudo apt install cockpit && sudo systemctl enable –now cockpit). This gets you a portal that can be accessed simply by navigating your browser to https://server-ip:9090. You login with your linux credentials and basically you get an interface you can use for managing and inspecting some basic services. Here’s the overview screen for one of my actual linux servers:
Cockpit comes pre-installed with several plugin ‘apps’ that expand and shape the functionality of the portal. You can add more plugins to make it even more useful (see my video) – e.g. for zfs service management:
You can even create virtual machines in cockpit that can be further viewed and configured with virt-manager. I showcase this in my video as I use cockpit on my real server to…create a virtual machine…in which I install and create a cockpit service and enable 2FA in the virtualized cockpit instance (!)
Whilst this is not as feature-rich as e.g. proxmox or xcp-ng, cockpit provides for a very capable and easy-to-use hypervisor in terms of the machines it creates.
There are lots of videos and tutorials about cockpit that can inform on its capabilities.
One concern I have about the basic installation for cockpit is that it gives you essentially the same access to a server as an ssh-connection without a public-private key: all it takes to login to a cockpit server is a username and a password. That’s too weak from a security perspective for server access IMHO. To fix that, we can add two-factor authentication (2FA) to the login, requiring the use of a 6-digit code in addition to the (potentially weak) user credentials. Here’s how you do that e.g. for a Debian/Ubuntu server (or you can watch my view above):
Login to your server, install cockpit and google’s authenticator app run the following command (as user, not root):
This updates your repositories, installs and enables cockpit, installs the authenticator app and runs it. After the installation, you will see an image similar to this as the 2FA app fires up:
Scan the QR code with your 2FA app, enter the code at the prompt and answer the questions to complete the process (answering ‘y’ is most secure, but even answering ‘n’ makes for a very secure 2FA setup – google-search if you want to know more). Now we need to tell cockpit to use 2FA: Edit the following file as root with your favorite editor, e.g nano:
sudo nano /etc/pam.d/cockpit
Add one line at the bottom of the file thus:
auth required pam_google_authenticator.so nullok
Save and quit, then issue the following:
sudo systemctl restart cockpit
Then login to your cockpit server, enter your linux username, password:
When you attempt to login, you get a separate chhalenge for your 2FA 6-digit credential:
Use your phone app to get the current 2FA code (which changes every 30 seconds), enter it and you should be logged in, e.g.:
There you have it, 2FA cockpit enabled. This is now much more secure just as it should be for linux server access.
Bonus: for those who access their ssh servers over WAN (I don’t), you can add 2FA access to your ssh connections too. I have an article here that shows you how to do that “more conveniently” than the typical ssh-2FA implementations.
TL;DR – Securely automate the decryption of your luks-encrypted Debian servers, to provide convenient yet substantial protection against data loss in the event of you losing some hardware. Here’s a video that goes through a real example of me encrypting a Debian 12 server using tang/clevis and showing the convenient yet substantially protective way it automatically decrypts my Debian servers on my home network:
The video clip includes me setting up two demo tang-servers in containers and using these servers to decrypt a luks-encrypted root file system on a Debian 12 server. All the instructions are in this blog post, but watching the video might make it easier for you to decide if this is for you. I encourage every Debian server user to luks-encrypt their drives to protect against data loss in the event of losing a disk (or even a whole server).
I ALWAYS luks-encrypt my linux drives, and I firmly believe you should too. Luks encryption is “full disk encryption”, and is typically used to protect against loss (including theft) of hardware. It ensures that your data are fully protected against physical hardware loss.
There are simply no exceptions to my rule of luks encrypting every drive I own. I do this because I don’t want my family’s data being accessed by anyone sleazy enough to e.g. break into my home and steal my hardware. It also makes it easy to dispose of a disk: disconnect if from power and put it in the trash. The contents are unreadable. No need to spend hours writing over each block of the disk to erase the contents. Quantum computing MAY make us revisit the impact of luks encryption, but today it’s still true: luks is solid.
Luks is a security measure and as such it can be “inconvenient” to use when it comes to boot disks and root file systems. To access luks-encypted drives you have to decrypt them, which typically means entering a password/key file to unlock them e.g. after power-up/reboot. For a desktop, that’s a completely acceptable task for me, but for bare-metal servers (and/or indeed virtual machine servers), this can be problematic.
There are TWO methods I use/have-used for decrypting luks servers: (1) dropbear-ssh (which allows you to gain ssh access to a luks-encrypted server during boot, so you can manually decrypt it) – this article is not about dropbear; (2) A combination of a tang-server and clevis package to automate that process to a large degree (the subject of this article).
There are many good articles on how to use dropbear-ssh and it’s pretty easy to do (google is your friend for researching this further). There are also tutorials on how to use clevis/tang. Most of these seem to be for Red Hat or maybe Ubuntu servers. I struggled a little to get clevis to work on my Debian 12 (Bookworm) servers, so I wanted to document how I do it. This is because I want to make it easy for YOU to luks-encrypt your Debian servers and thus strongly protect from equipment-theft based threats.
This article is however a little more targeted in that it also intended for someone who has MORE THAN ONE physical server. If you only have one, I am not sure it’s really worth having a tang/clevis system, since your services go offline during a re-boot no matter what. I personally have THREE physical servers in my home-lab (because I am a little very crazy), and I work hard to make sure at least one of them is always up. So I configure clevis/tang to restart any two servers that get power-cycled. If all of my servers get power-cycled at once, they cannot automatically decrypt via clevis. I designed it that way deliberately.
So how does tang/clevis (two different software packages), work together to decrypt a luks header “automatically”? Well there are many excellent articles on that subject (e.g. here, and here, so please check them out), but in summary form, here’s what you do to use it, and a simplified version of how it works:
Install Debian server via an iso installer onto your hardware (or virtual machine) as usual, but during the disk partitioning screen, select the option for ‘Guided, use entire disk and setup encrypted lvm’ during the partitioning screen, e.g. for the net-install method for Debian12:
Enter a (very strong!!!) password at the ‘encryption passphrase’ prompt. This should use upper & lower case, symbols and numbers and should be awkwardly long to make brute-force impossible. For even better security, DO NOT USE A PASSWORD MANAGER, MAKE IT SOMETHING YOU CAN REMEMBER. NEVER WRITE IT DOWN, (Warning: if you forget this, bad things will happen to your data in terms of your ability to ever read them again!!).
Complete the rest of the installation as normal, which ends with a reboot.
Your first test: enter the luks encryption password when prompted to decrypt the OS root file-system and allow it to boot-up.
Your server is now ready to have clevis installed and be configured to auto-decrypt your luks root file-system.
HOW DO I USE TANG/CLEVIS?
Before we start typing commands, here’s how I actually run my tang-clevis setup: I actually run TWO tang services, one of which is remote (not on my LAN, but still under my (remote) control), and the other does run on my LAN. In fact, the LAN-based tang-server is a high-availability cluster that can provide a single (identical) tang service to any machine on my LAN as long as at least one of the tang servers is online.
So my bare-metal servers have to get blinded decryption keys from two separate services for them to auto-decrypt at boot. One of which is provided by my highly-available tang server on the LAN, the other tang server is, well, somewhere else. This setup allows me to simultaneously reboot any TWO of my three LAN-based servers and they will automatically decrypt and reboot IF both tang services are available.
This is a deliberately fragile service: if anything is up with a tang service, chances are I have to manually decrypt my servers. In practice, I rarely do, so this just quietly runs and does its thing. I like that. Worse case, I can always vpn into my LAN, access a server via the IPMI interface and manually enter a luks code, but I am safe if e.g. anyone steals my hardware: any one of my physical servers that goes offline will NOT boot up if ANY of the following conditions are true:
All servers have been power-cycled simultaneously (e.g. switched off/power-cut…or stolen!)
Any server is disconnected from my local network prior to being rebooted (e.g. a thief back at their evil headquarters)
If my remote tang-server goes offline for any reason (more on this below, because sometimes I make it go offline deliberately)
And of course all the other reasons why services can fail…all of which are usually unintended of course!
I like all this, it allows me to sleep safe. I may lose my hardware, but I won’t ever lose our family data to a thief who steals my hardware (this method offers no protection against online hackers by the way – keep good offline backups!!).
For my remote tang-server, I am also experimenting with a systemd service that I have written, that does one extra thing: it tracks the status of my LAN servers to see if they are online. If they are, it keeps the remote tang-server hot and ready to serve blinded keys to them. If a LAN server goes offline, the systemd service sets a timer – enough time to allow for an automated reboot. If the server remains OFFLINE, for any reason, beyond my timer (e.g. in a scenario where someone has stolen my hardware, taken it somewhere and is attempting to power it up to see if they can somehow get my tang keys), the remote tang server is taken offline and the system it is on is rebooted (and thus deliberately locked via luks). No more remote tang-server until and unless I manually bring it back online. This extra complication essentially protects against even sophisticated thieves trying to steal a single server and use the online ones to somehow get to my data. If all goes well (and frankly it pretty much always does), my LAN servers automatically come back up after a reboot, but if any fragility kicks-in, I am there to do it manually. Very cool. As I said, I may lose my hardware to thieves, just not my data.
TECHNICAL DETAILS
We need to create a tang server (or better yet two or even more) to provide a tang service, and we need to install and configure clevis on each of the Debian servers we have luks-encrypted (in my case, that means all my bare metal servers). And then we need to link our tang services to a physical server’s boot luks header. Below, I will go through the simple steps needed to complete all of this.
Pre-requisite
I am assuming you have two or more Debian luks-encrypted servers that you want to configure clevis on. You need root access to these and details of IP address and basic networking (network gateway IP etc.).
I also assume you have a means of running a container on these servers, so we can install the tang services. These tang services have to be online, available and accessible to the luks-encrypted Debian server, so that at boot time the encrypted server can access these tang services on at least one of the other LAN tang servers in order to auto-decrypt the luks-encrypted root file system. In practice, this gives you a LOT of options (my own setup is a more complicated derivative of this), but I will try to keep it simple below.
Tang Server(s)
Tang is a service, like e.g. a web server. It just sits there listening at a port for a request to encrypt/decrypt a phrase. So it’s easy to setup. Just spin up your preferred method of virtualization (e.g. an incus or docker/lxd/lxc container, or a vm if you prefer). Ideally, this virtual instance should be installed on all of your encrypted servers so that it can provide a tang service on your LAN if any one of your servers is online. This is not essential, but I feel it’s a more secure way to implement tang/clevis.
Create and login to the soon-to-be tang server. You can use as many different servers as you like (I use two in this example) to create two or more tang servers. Per below, just enter and execute one command with root privileges:
sudo apt install tang jose
Hit enter, confirm the installation and let it complete. That’s all you need to do. Note down the IP address of the container and exit the container.
We can check that the tang service works by using a curl command. In the example below, I am assuming your IP address is 10.231.25.130. Change to suit:
curl http://10.231.25.130/adv
Just enter the above curl command (with the correct IP address) into a terminal and hit enter. Assuming you can connect to the server (i.e. on the same network) then the image below resembles what you should see (I use echo commands before and after, just to give some spacing in the output):
If you get a long string of similar looking “visual garbage” then…it’s working as it should! And that’s it. One tang server, ready to go. I recommend creating one on your LAN, and also one “somewhere else” (a distant friend/family’s computer, a hosting company that sells/rents vps’ etc. – somewhere pretty much out of the way). It’s not essential, it’s just how I do it. Once you set it up, just issue a ‘curl http:<ip-address>/adv’ command to fetch the advertising key per the above to see that it’s accessible as I show above. That’s it, one tang server ready to work for you (or indeed several of them).
[To make this even more secure, I recommend that you place at least one of these tang servers in a clustered high-availability mode on your network of encrypted physical servers. Doing so does mean that “at least one” of your LAN servers has to be online for this to work, which for most operations should be fine. If anyone wants to know more about this, just let me know (e.g. via @OGSelfHosting on X)]
Installing and Configuring Clevis
So now, let’s configure a luks encrypted server to coordinate with the tang services.
Login to your luks encrypted Debian server and enter a few commands as follows. Firstly, install clevis and it’s needed plugins:
For this next step we need networking information specific to your LAN. I am assuming you have TWO tang servers that you want to bind against a clevis key, setup as I showed earlier. I further assume these are accessible via the following ip addresses (change IP addresses to your tang servers):
When you use two servers, as I have done here, the clevis command uses a ‘shamir secret sharing’ (the ‘sss’ option), which basically instructs clevis to use two (or more) tang-servers to create a fragmented blinded luks key. If one of these tang-servers is on your LAN (especially on one of your encrypted servers), and one is in a distant location, this makes the service very very secure. Probably as good as it can be given that this is “automated”.
When you enter the above command, clevis asks for your luks decryption password (enter it accordingly), then it checks the tang servers and asks if you want to trust them (review and confirm). Clevis does some checks on the data and if there are no errors, it creates a new luks header key using a key derived from numbers sent back from the two tang-servers via a clevis-initiated prompt. Note that neither tang-server knows the luks password, they merely return a fragmented blinded code which on its own cannot decrypt a luks disk. Only clevis (and luks) know this password. Your old luks password WILL STILL WORK AS NORMAL but now there’s a new key that clevis can use to unlock the root partition at boot. But we are not done yet, we have to do some configuration so that there is network access during boot-up. Edit the following file on your Debian server:
sudo nano /etc/initramfs-tools/initramfs.conf
The initramfs file allows you to specify information about your network. if you google initramfs you can find a lot more information, e.g. here. We are going to specify some of our networking – just enough to get an IP address on the LAN and internet access. Change these to suit your setup, then add this line at the end of the above file and save/quit. The items in bold will likely need changing. The network mask as shown below works for a /24 network (likely good enough for most people), but if you are running something more sophisticated then change this too.
The IP config has to be right for clevis to work (but you can always edit this file after a boot, so it’s not world-ending if you make a mistake). Do not miss out the single and double colons. Google-search initramfs if you want to know more, but basically this sets ups most of the networking needed for clevis to go find a tang server. Save and quit the config file change when done. We have one more file to create, to allow us to potentially connect to remote tang servers (if required):
sudo nano /usr/share/initramfs-tools/hooks/curl
Paste the following commands into the file (which I got from this link). It provides a name-server capability during boot then save/quit (so that tang can convert e.g. “remote-tang.mydomain.com” into an actual IP address), which you need if a tang-server is actually located on another network (i.e. remote):
#!/bin/sh -e
PREREQS=""
case $1 in
prereqs) echo "${PREREQS}"; exit 0;;
esac
. /usr/share/initramfs-tools/hook-functions
#copy curl binary
copy_exec /usr/bin/curl /bin
#fix DNS lib (needed for Debian 11)
cp -a /usr/lib/x86_64-linux-gnu/libnss_dns* $DESTDIR/usr/lib/x86_64-linux-gnu/
#fix DNS resolver (needed for Debian 11 + 12)
echo "nameserver 1.1.1.1\n" > ${DESTDIR}/etc/resolv.conf
#copy ca-certs for curl
mkdir -p $DESTDIR/usr/share
cp -ar /usr/share/ca-certificates $DESTDIR/usr/share/
cp -ar /etc/ssl $DESTDIR/etc/
Now save the file/quit, make it executable then update the initramfs image. Then reboot as we are all DONE:
sudo chmod 755 /usr/share/initramfs-tools/hooks/curl
sudo update-initramfs -u -k 'all'
sudo reboot #This is optional, but it's how we test it. :)
If everything works as expected, you will watch your server boot up, ask for a luks key, enable networking (get an IP, connect to the adapter etc.) then it will try and contact the tang servers. Assuming success, watch the service magically unlock your encrypted server drive and for the system to boot up as if you entered the luks key yourself. It’s a satisfying thing to watch!
You can stop a tang server to disable auto-luks decryption, and that is an extra layer of protection you can further configure to make this convenient yet substantially protective.
if you have avoided luks encryption because of the inconvenience of having to connect to your server every time you reboot, PLEASE think again. Tang and clevis make this super convenient yet I believe also extremely secure.
If there’s any interest, I’ll update on my ‘systemd’ optimization on my remote server, which I think makes this “better than a human” at protecting my luks-encrypted Debian servers during reboot, making it extremely particular about how a server can actually reboot and auto-decrypt.
I hope you found this tutorial useful. Message me if you have any questions or comments, e.g. at @OGSelfHosting on X.
