[Guide] Fedora 36+: Hibernation with enabled secure boot and full disk encryption (fde) decrypting over tpm2

This is a somehow personal step-by-step documentation, how I achieved hibernation and suspend-then-hibernate on a recent Fedora system with enabled secure boot. Basically you patch the kernel to allow hibernation with enabled secure boot and then configure hibernation. This can only be done securely, because we use a swap file stored on an encrypted disk. I used following documentation snippets on the internet:

1. Enable automatic decryption of luks over tpm

I have a standard Fedora installation with full disk encryption. To enable automatic decryption over tpm2, you add necessary files to dracut and add the decryption key to the tpm module. You basically follow these instructions.

# Add decryption key to tpm. 
systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+2+4+7 /dev/nvme0n1p3

# Wipe old keys and enroll new key. You have to execute this command again after a kernel upgrade.
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=0,2,4,7

# Add tpm2 configuration option to /etc/crypttab
luks-$UUID UUID=disk-$UUID none tpm2-device=auto,discard

# Update initramfs (to get necessary tpm2 libraries and parameters for decryption into initramfs)
dracut -f

2. Patch and compile custom kernel

You follow basically this how-to. For a simplification I just wrote down the commands and variables I used for the most recent kernel on a Fedora 37 system.

### Set variables
export arch=x86_64
export ver=6.0
export minrel=15
export pkgrel=300
export subver=$minrel-$pkgrel
export fedver=fc37
export name=$(hostname)

### Custom Machine owner key for secure boot
# Allow kernel signing
sudo /usr/libexec/pesign/pesign-authorize
# Create key
openssl req -new -x509 -newkey rsa:2048 -keyout "key.pem" -outform DER -out "cert.der" -nodes -days 36500 -subj "$name"
# Import key to UEFI database.
mokutil --import "cert.der"
# You have to reboot the system after importing the key with "mokutil" to import the key via UEFI system
# After rebooting create PKCS #12 key file and import it into the nss database
openssl pkcs12 -export -out key.p12 -inkey key.pem -in cert.der
certutil -A -i cert.der -n "$name" -d /etc/pki/pesign/ -t "Pu,Pu,Pu"
pk12util -i key.p12 -d /etc/pki/pesign

### Setup build system
rpmdev-setuptree
koji download-build --arch=src kernel-$ver.$subver.$fedver
rpm -Uvh kernel-$ver.$subver.$fedver.src.rpm
cd ~/rpmbuild/SPECS

### Apply patches and customize kernel configuration
# Get patch to enable hibernate in lockdown mode (secure boot)
wget https://gist.githubusercontent.com/kelvie/917d456cb572325aae8e3bd94a9c1350/raw/74516829883c7ee7b2216938550d55ebcb7be609/0001-Add-a-lockdown_hibernate-parameter.patch -O ~/rpmbuild/SOURCES/0001-Add-a-lockdown_hibernate-parameter.patch
# Define patch in kernel.spec for building the rpms
# Patch2: 0001-Add-a-lockdown_hibernate-parameter.patch
sed -i '/^Patch999999/i Patch2: 0001-Add-a-lockdown_hibernate-parameter.patch' kernel.spec
# Add patch as ApplyOptionalPatch
sed -i '/^ApplyOptionalPatch linux-kernel-test.patch/i ApplyOptionalPatch 0001-Add-a-lockdown_hibernate-parameter.patch' kernel.spec
# Add custom kernel name
sed -i "s/# define buildid .local/%define buildid .$name/g" kernel.spec
# Add machine owner key
sed -i "s/.$name/.$name\n%define pe_signing_cert $name/g" kernel.spec
# Install necessary dependencies for compiling hte kernel
rpmbuild -bp kernel.spec

### Optional steps
# Create own configuration file from fedora config file
# You find my "minimized" configuration for a 6.0.11 kernel here.
cp ~/rpmbuild/SOURCES/kernel-$arch-fedora.config ~/rpmbuild/BUILD/kernel-$ver.$minrel/linux-$ver.$subver.$name.$fedver.$arch/.config
cd ~/rpmbuild/BUILD/kernel-$ver.$minrel/linux-$ver.$subver.$name.$fedver.$arch/
make menuconfig
# Copy custom menuconfig kernel configuration to kernel-local
cp ~/rpmbuild/BUILD/kernel-$ver.$minrel/linux-$ver.$subver.$name.$fedver.$arch/.config ~/rpmbuild/SOURCES/kernel-local
# ... or copy kernel config from running kernel to kernel-local
#cp /boot/config-$(uname -r) ~/rpmbuild/SOURCES/kernel-local
# Remove build infos from custom config
sed -i '0,/^#\ General\ setup$/d' ~/rpmbuild/SOURCES/kernel-local
sed -i '1i # x86_64' ~/rpmbuild/SOURCES/kernel-local
### End optional steps ###

# Compile kernel
cd ~/rpmbuild/SPECS
time rpmbuild -bb --with baseonly --without debuginfo --target=$arch kernel.spec | tee ~/build-kernel.log

# Install kernel
cd ~/rpmbuild/RPMS/$arch/
sudo dnf install *.rpm

After installing the new kernel and before rebooting, you have to add a kernel parameter to allow hibernation in kernel lockdown mode (actually allow hibernation with secure boot).

grubby --args="lockdown_hibernate=1" --update-kernel=ALL

Now reboot your system.

3. Enable hibernation

Follow this excellent article on fedoramagazine.org to enable hibernation: Hibernation in Fedora Workstation - Fedora Magazine. With this, you create a swap file on the encrypted hard disk which will be started and removed upon hibernation or waking up.

4. Preparation for suspend-then-hibernate

The suspend-then-hibernate doesn’t work out of the box with this setup (swap file for hibernation on luks encrypted hard disk). The system needs to prepare and remove the swap file before and after executing suspend-then-hibernate, check out this excellent gist: Fedora 35 hibernation with swap file, only for hibernation and resume · GitHub

5. Enable suspend-the-hibernate

Normally, when you close the lid, the system will go to sleep (a.k.a. “suspend”). You can configure your system that it will hibernate automatically after an extended sleep. Check this excellent guide here.

Final words

All in all it is quite complicated to get a “secure” system with enabled secure boot and smoothly working hibernation, but in my case, it saves a lot of battery … As a plus you may enable encrypted RAM in the bios settings.

18 Likes

Pinning this for a couple of weeks. This is great!

Mods, if you have a better way of making this visible, please feel free to adjust - but make sure this remains visible please. :slight_smile:

Big thank you for this guide :pray:

I still have some questions :

  1. Why have you decided to use secure-boot ? I currently have a fully encrypted system with suspend-then-hibernate working. I just don’t have secure-boot and I’m quite interested with your pros and cons.

  2. Do you think the kernel patch will be integrated upstream one day ?

  1. Because without secure boot it wouldn’t be able to decrypt the hard disk with the TPM chip (First I am a lazy person and I don’t want to enter my long decryption passphrase for luks every time I boot the system and second I d’like to boot my system remotely via wake-over-lan without ssh in initramfs to decrypt luks.)
  2. I don’t know, but I really hope so … or that there will be at least something similar someday in the kernel. Afaik there is still a lot of work going on in the hibernation-power-management thing in the kernel.

As I am not a lazy person, I am satisfied with my own secure boot+FDE with a detached LUKS header and a passphrase, no tpm or kernel compiling required. To each their own.

Nice guide :slight_smile:

One question:

Was this step still actually needed? That Github gist (and the bugzilla ticket it links) seems to indicate that for Fedora 36+ (or at least dracut 056+), it’s not an issue anymore.

Edit:
Can confirm that I didn’t need to make those changes. I did have to run dracut -f once manually, but then dracut --list-modules | grep tss showed that tpm2-tss was being included in my initrd. My very rough guess skimming that bugzilla ticket is that by editing /etc/crypttab, the next time dracut is run, it goes oh, I need TPM support in the initrd now.

Similarly, I didn’t need to append the rd.luks.options=tpm2-device=auto kernel argument. Pretty sure LUKS is already figuring that out from the option set in /etc/crypttab but :person_shrugging:

It seems you’re right. I’ll adapt the guide.

I’m having problems with pesign. Here are the error messages.

During the final steps of command time rpmbuild -bb --with baseonly --without debuginfo --target=$arch kernel.spec | tee ~/build-kernel.log:

/usr/bin/pesign-client -t ‘OpenSC Card (Fedora Signer)’ -c framework -s -i arch/x86/boot/bzImage -o vmlinuz.tmp
pesign-client: could not get response from server: Connection reset by peer
error: Bad exit status from /var/tmp/rpm-tmp.1nHlOb (%build)
Bad exit status from /var/tmp/rpm-tmp.1nHlOb (%build)

output of journalctl -xeu pesign:
Jan 17 15:27:25 framework pesign[115605]: searching for command 7
Jan 17 15:27:25 framework pesign[115605]: cmd-version: found command “sign-attached-with-file-type” version 0
Jan 17 15:27:25 framework pesign[115605]: unlock-token: invalid data
Jan 17 15:27:25 framework pesign[115605]: possible exploit attempt. closing.
Jan 17 15:27:25 framework pesign[115605]: unlock-token: invalid data
Jan 17 15:27:25 framework pesign[115605]: possible exploit attempt. closing.
Jan 17 15:27:25 framework pesign[115605]: attempting to sign with key “OpenSC Card (Fedora Signer):framework”
Jan 17 15:27:25 framework pesign[115605]: cms_common.c:find_certificate:625: Could not find token “OpenSC Card (Fedora Signer)”: Failure to load dynamic library

output of modutil -dbdir /etc/pki/pesign -list:

Listing of PKCS #11 Modules

  1. NSS Internal PKCS #11 Module
    uri: pkcs11:library-manufacturer=Mozilla%20Foundation;library-description=NSS%20Internal%20Crypto%20Services;library-version=3.87
    slots: 2 slots attached
    status: loaded

    slot: NSS Internal Cryptographic Services
    token: NSS Generic Crypto Services
    uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

    slot: NSS User Private Key and Certificate Services
    token: NSS Certificate DB
    uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

  2. p11-kit-proxy
    library name: p11-kit-proxy.so
    uri: pkcs11:library-manufacturer=PKCS%2311%20Kit;library-description=PKCS%2311%20Kit%20Proxy%20Module;library-version=1.1
    slots: There are no slots attached to this module
    status: loaded

#List of modules in pesign database:
certutil -d /etc/pki/pesign -U

slot: NSS User Private Key and Certificate Services

token: NSS Certificate DB
uri: pkcs11:token=NSS%20Certificate%20DB;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

slot: NSS Internal Cryptographic Services

token: NSS Generic Crypto Services
uri: pkcs11:token=NSS%20Generic%20Crypto%20Services;manufacturer=Mozilla%20Foundation;serial=0000000000000000;model=NSS%203

After reading Fedora_RPM_pesign.md · GitHub

I modified the rpmbuild command to:

time rpmbuild -bb -D ‘pe_signing_token NSS Certificate DB’ -D ‘pe_signing_cert framework’ --with baseonly --without debuginfo --target=$arch kernel.spec | tee ~/build-kernel.log

It worked. I don’t know why rpmbuild defaults to token ‘OpenSC Card (Fedora Signer)’ though.

I’m curious how you got to level 3 in security. My default fedora install with secure boot stops at level 0 because Intel ME is disabled.

I have one other question. Since your’e using TPM to handle luks encryption, what happens if the device breaks? Can you backup the decryption keys so you can remount the drive in another device and recover the data?

How?

You can set up a recovery key using systemd-cryptenroll or choose not to wipe the passcode set when LUKS encrypted the drive. The man page has more details

1 Like

At least on (some) 11th gen models (not sure about 11th gen 1185g7), ME firmware needs to be upgraded with a BIOS update. Beyond that, only vPRO models can handle encrypted RAM/etc for level 3 afaik

1 Like

This is fantastic, thank you so much! :cowboy_hat_face:

One question: Given you have to patch the kernel, what do you do when a new version is available? Do you just avoid updating the kernel, or is there a streamlined approach to make the patch persist between upgrades?

No, not yet. I just recompile the kernel with the patch and install it. Actually, I just run these commands. With my streamlined kernel configuration, the compiling takes about half an hour on an 12th generation board.

### Set variables
export arch=x86_64
export ver=6.1
export minrel=7
export pkgrel=200
export subver=$minrel-$pkgrel
export fedver=fc37
export name=$(hostname)

### Setup build system
rpmdev-setuptree
koji download-build --arch=src kernel-$ver.$subver.$fedver
rpm -Uvh kernel-$ver.$subver.$fedver.src.rpm
cd ~/rpmbuild/SPECS

### Apply patches and customize kernel configuration
# Get patch to enable hibernate in lockdown mode (secure boot)
wget https://gist.githubusercontent.com/kelvie/917d456cb572325aae8e3bd94a9c1350/raw/74516829883c7ee7b2216938550d55ebcb7be609/0001-Add-a-lockdown_hibernate-parameter.patch -O ~/rpmbuild/SOURCES/0001-Add-a-lockdown_hibernate-parameter.patch
# Define patch in kernel.spec for building the rpms
# Patch2: 0001-Add-a-lockdown_hibernate-parameter.patch
sed -i '/^Patch999999/i Patch2: 0001-Add-a-lockdown_hibernate-parameter.patch' kernel.spec
# Add patch as ApplyOptionalPatch
sed -i '/^ApplyOptionalPatch linux-kernel-test.patch/i ApplyOptionalPatch 0001-Add-a-lockdown_hibernate-parameter.patch' kernel.spec
sed -i "s/# define buildid .local/%define buildid .$name/g" kernel.spec
sed -i "s/.$name/.$name\n%define pe_signing_cert $name/g" kernel.spec
# Copy currently running, customized and streamlined configuration (see first post how to customize)
cp /boot/config-$(uname -r) ~/rpmbuild/SOURCES/kernel-local
# Remove build infos from custom config
sed -i '0,/^#\ General\ setup$/d' ~/rpmbuild/SOURCES/kernel-local
sed -i '1i # x86_64' ~/rpmbuild/SOURCES/kernel-local

### Compile kernel
cd ~/rpmbuild/SPECS
time rpmbuild -bb --with baseonly --without debuginfo --target=$arch kernel.spec | tee ~/build-kernel.log

### Install kernel
cd ~/rpmbuild/RPMS/$arch/
sudo dnf install *.rpm

The needed patch worked with kernel version 6.0 and still works with 6.1. I hope, the author will adapt it when needed so, otherwise I’ll probably just adapt it myself (and post it here).

1 Like

This is great! Thank you so much for all this documentation. I’m going to try setting this up this weekend.

I updated my personal kernel configuration (version 6.1.9). Now it takes about fifteen minutes to compile the kernel. Check out my streamlined kernel configuration for frame.work. Copy this file to ~/rpmbuild/SOURCES/kernel-local to get a streamlined kernel for the frame.work notebook. You probably want to customize/enable more device drivers depending your needs …

Amazing work ! Thank you so much for your time !
Are we going to have to perform these steps on every kernel version or is the fedora project going to add/work on these to the next kernel version ?

@Raining-characters I don’t think they are going to work on it at all. This is all a hack to circumvent kernel lockdown. As far as Fedora is concerned, the lack of working hibernation means everything is working as intended.

1 Like