Proxmox Backup#

I am a long time Proxmox user, the earliest use of it on this site is sometime in 2015 though my use predates that. However like many people within the industry of IT, we’re migrating away from VMware to other platforms. Infact I did this just over a year ago now with my day job (and rather successfully I may add).

This however does leave a small issue of Backup. Unlike VMware where to get the most out of backup you need a third party tool (we used Veeam), Proxmox can do it’s own backups directly to an external drive, network mount or well anything you can path on the host machine. However the best option is a first party tool in the form of Proxmox Backup Server.

It launched as a fairly featured tool however has gotten much better as time has gone on, it’s ease of use and direct integration into Proxmox with very little hassle is simply great.

Why Encrypt the Backups?#

If you’re in business, it is usaully a compliance and audit requirements.

If you’re a homelabber, it’s not so much a requirements. But more importantly it keeps your data safe if you say then back it up to cloud or a remote machine.

Encrypting Backups#

This is easy, honestly it is. Proxmox saw this coming and it is built right into the mounting options for PBS (Proxmox Backup Server) within PVE (Proxmox Virtual Environment).

If you’re happy to work with what they provide (according to their docs at time of writing that is AES-256-GCM PBS Docs), it will even generate the keys for you! PVE Encryption Option on Drive Mount

Please for the love of all that is Holy, keep these keys safe. Although restoring VM/CT backups into the same host is easy (or if a cluster shares the same key), there maybe a day you need to disaster recover and restore to a different machine from scratch.

PVE, if you ask it to generate the keys, will throw these same warnings. PVE Key safe keeping warning

In the case of work, I have these printed out and copied on to USB drive in a sealed envelope at both our Main and Disaster Recovery site. In the popular adage “Failing to prepare, is preparing to fail”.

Please print them as well!!!! Leaving a USB drive unplugged for some time can lead to data being lost. Simply due to the laws of physics. This may take years, but better safe than sorry.

Now that all that doom and gloom is out of the way, you can start creating encrypted backups!!!! Woooo, Hits the ‘That was Easy’ Button

Encrypted Backup Pitfalls#

The only problem you have now is that you’re backups can only be worked upon with the encrpytion key. As PBS does NOT have the key you encrypted with, it does mean viewing your Backups do look slightly different.

No Encryption PBS options

With no Encryption, you can download all the files in the backup directly from PBS. You can even download specific files out of the backed up disk image.

Encrytion PBS options

However, with encryption enabled you’re now limited to just the client.log.blob and index.json.blob. You can still do a ‘File Restore’ from PVE (as it has the keys), but not within PBS.

You can still recover these without a full PVE install, I will go over that further down.

I LOST THE KEYS#

Well if you have the PVE still running, then they are stored at:

/etc/pve/priv/storage/

Otherwise, SOL.

Or the keen eyed amoung you may have noted the ‘rsa-encrypted.key.blob’ in the last screenshot… Jump to ‘Making it better’

Restoring Backups#

So if you are going back into PVE from PBS, this is not an issue. You will see no difference, it may take slightly longer due to the decryption, but other than that from PVE perspective, nothing has changed.

Mounting PBS without the correct key#

If you mount PBS drive in PVE without the correct key, it will mount and it will show you your backups, however…

Missing Key warning

The missing key warning will greet you and you will not be able to recover.

Remounting PBS on a different PVE machine#

The backup encryption is hinged upon the .enc file that you WILL HAVE KEPT SAFE.

If you have the digital .enc file, at time of mounting the PBS drive you will simply upload the key.

Upload Key to PBS

What if it’s all gone belly up and you only have the paper version?

The paper version will look like the below:

Paper Version Example This is just an example, not an active key. Yes I blacked it out as some idiot or AI will try and read it.

You will have a chunk of text at the top, this is essentially the contents of that .enc file. You can manually type that out into a text document without anyline breaks and save it as .enc and it will work.

The QR code is just that text blob, but hopefully will save you typing it out.

Once you have that back in a digital format, then just upload it like previously mentioned.

Recovering Data without PVE#

What if somewhere down the line you have the backup files, you have the key but you no longer have access to PVE or PBS.

taps sign “Failing to Prepare, is Preparing to Fail”

You will at the bare minimum need the Proxmox Backup Client, this has it’s own repo and will need to be installed. If we’re covering off the ‘What if Proxmox no longer exists’ scenario, you may wish to take a backup of the .deb file for Proxmox Backup Client.

This will be covered off in the Backing Up Standalone Machines to Proxmox Backup Server

But is this as good as it can be?#

Well yes and no. Yes in terms of the encryption you can use, no in terms of encryption key security.

As mentioned above below one of the PBS encryption screenshots, I noted the ‘rsa-encrypted.key.blob’ in the backup. When you use a Key pair within PVE for the backup, it will use the public key to encrypted the .enc file and place it with the backup.

The keypair is only used to encrypted the .enc file and no other part of the backup. I mean it would be nice to have the keypair do that, but as of writing this is not possible.

Making it better!#

First off you will need a key pair for PVE to use.

This can be done within PVE itself with the following command:

proxmox-backup-client key create-master-key

By default this makes a 4096-bit RSA key. I am not sure if PVE will use something different like EDCSA-384, so I will leave that up to you to discover. It will prompt for a password to unlock the private key so please set something A: Secure and B: you will remember.

This will then create the following two files:

master-private.pem  master-public.pem

Copy the private key off of the server, keep it safe this is your failsafe to your other backups of .enc being inaccessible. I would then remove it from the server. It’s one of those things of if people are deep enough into your environment, they already have access to the .enc and anything on the host. Its more of a precaution.

You will then need to rename the ‘master-public.pem’ to:

{name of mount}.master.pem

For example my PBS is mounted as PBS so therefore my key is called:

pbs.master.pem

You will then need to move this to the following path:

/etc/pve/priv/storage/

You will also find the .enc and a .pw file. These are the encryption key and weirdly the password for the PBS mount?

Once in there you will need to modify the storage config so that it recognises there is a master public key to use.

Open the following in your editor of choice:

/etc/pve/storage.cfg

Locate the section for your PBS mount. If you only have one (with encryption on), then look for the entry with a ’encryption-key’ line. You will then need to add:

master-pubkey 1

And you’re done!

Now if you do a backup, your backup will be encypted and have a separately encrypted file with the encyption key in.

Recovering the .enc with the private key#

To do this you will need to get the private key back on to the PVE host (or machine with Proxmox Backup Client). Also you will need the ‘rsa-encrypted.key.blob’ file on the machine.

Then all you need to run is the following:

proxmox-backup-client key import-with-master-key ./decrypt --master-keyfile ./master-private.pem --encrypted-keyfile ./rsa-encrypted.key.blob

This will both decrypt the .enc file and copy it to the hidden .config file within the users home path. From there copy it to where it is needed (like the ‘/etc/pve/priv/storage/’ path on PVE).

Side note for the .blob files#

THIS DOES NOT APPLY TO THE ‘rsa-encryted.key.blob’

If you ever need to read the index.json.blob or client.log.blob from the backup repo, you need to uncompress it. However it has a custom 12-byte header, so you will need to remove it:

dd if=/path/to/blob bs=1 skip=12 | zstdcat

And that will now make the Blob not so bloby and readable.

Next Steps#

Print those enc files, keep them safe. Do the same with digital versions. Keep the private key from the generated key pair (if you also did that) safe somewhere.

And enjoy the now encypted backups.

Consider backing up non-virtual machines to PBS or even modifying settings to make your backups immutable.