Backups you can restore

Anyone who has worked with backups has seen it: a console full of green ticks, and then the day something actually needs restoring and it can't be done. Maybe a disk is missing from the job. Maybe the database is inconsistent. Maybe the restore takes twelve hours when the business was counting on one.

A backup is not the goal. The goal is being able to restore. That sounds like wordplay, but it changes what you measure and what you test.

Start from the other end: what has to come back, and how fast?

Before talking about software and storage, you need two answers for each system:

• How much data can we afford to lose? (RPO) A night's worth? An hour's? None?
• How long may the system be down? (RTO) A working day? An hour?

The answers don't come from IT. They come from the people who own the systems and the business. When I have done this work, it has often turned out that a single standard policy protects some systems too much and others too little. Sitting down with the application owners and going through the systems one by one is tedious, but it is where the protection level gets set right.

Application-aware backup for anything with state

A virtual machine running a database can't just be snapshotted on the fly and be expected to work after a restore. The database needs to know the backup is happening, so that it reaches a consistent state and the log chain is handled correctly. For SQL Server that means application-aware backup, and in clusters with availability groups you also need to check that the backup doesn't break the log chain of the regular database backups.

You can check this. The database's own history shows which backups were taken and whether they were marked as copy-only. I don't trust that the backup console shows it correctly; I check in the database.

Make the backup immutable

Ransomware goes looking for backups first. If an attacker with administrator rights can delete them, they aren't worth much. So at least one copy should sit on storage where it can't be changed or deleted for a set period, not even by an administrator:

• A hardened Linux repository with immutable files, no SSH and minimal access.
• Object storage with object lock.
• An off-site copy, in another datacentre or with another provider.

The old 3-2-1 rule still holds: three copies, two different media, one off-site. Today you add a one and a zero: one copy that is immutable or offline, and zero errors when verified.

The storage under the backup matters

Backup software works with large blocks and fast clones of existing data. If the file system under the repository is formatted with the wrong cluster size, it can work for months and then start throwing errors or performing badly as the volumes grow. It is an example of a fault that doesn't show in the daily status, but decides whether the restore works on the day you need it. Check the vendor's recommendations when the repository is created, and document how it was done.

Test the restore, for real

The only way to know whether a backup works is to restore it. It doesn't have to be dramatic:

• Regularly and automatically: let the backup system boot machines in an isolated network and check that they respond.
• Manually, a few times a year: restore a whole system, for example a test environment for a production system, and time it. Does it match what the business expects?
• Single files: the most common restore case. Test that too.

After a larger test I write a runbook and a short test report. The runbook is for whoever does it next time, possibly in the middle of the night. The report shows what took time and what needs to change.

Don't forget the keys

Encrypted backups are the right call. But if the encryption key, the password for the backup software or the login to the storage only exists on the server that just burned down, the backup is unreadable. Where are the passwords? Who can get to them? Has anyone tried restoring from scratch, on a new machine, with nothing but what is in the password manager?