When the power goes

When the power goes in a factory datacentre, two things happen at once. The UPS takes over, and a clock starts ticking. How much time you have depends on the batteries, the load and how old the batteries are. What happens during that time depends entirely on how well you prepared.

A UPS buys minutes, not safety. If the servers are still running when the batteries run out, everything goes down hard, all at once, in the middle of writes. That is how you lose data, end up with corrupt databases and get a cluster that refuses to start again.

Know how much time you really have

The runtime quoted for a UPS applies to a specific load and new batteries. Reality is a different load and batteries that have aged. So you need to:

• measure the actual load, rather than adding up the maximum ratings on the nameplates,
• allow a margin for battery ageing,
• include everything that has to stay up for the shutdown to work: network switches, firewall and management servers.

If the network dies before the servers, the shutdown commands can't get through. Counting only the servers is a common mistake.

Graceful shutdown in the right order

The point of graceful shutdown is that software, not the battery, decides when and how things are switched off. An agent or management software talks to the UPS and runs a plan once power has been out for a set time or the battery reaches a threshold.

In a virtualised environment, order matters:

1. Application virtual machines shut down first, in an order that respects dependencies. Application servers before databases.
2. Infrastructure services such as directory services and management go last among the virtual machines.
3. The cluster is prepared, so that storage is in a consistent state.
4. The hosts power off.

The order should be written down and tested, not something you hope the software gets right.

Clusters across several locations need several scenarios

In a cluster where the nodes sit in two different places, perhaps two buildings within the same factory with a third point acting as witness, a power cut is not a single scenario. It can be:

• one location losing power,
• the other location losing power,
• both at once.

If only one location loses power, you usually do not want to shut everything down. The cluster is built so that the other location takes over. The shutdown logic therefore has to understand the fault domains and only switch off what is genuinely at risk. That is where a simple "shut everything down at 50 per cent battery" gets it wrong.

Working out that logic takes time, and it has to be done together with people who know both the power supply and the applications. But it is the difference between a cluster that rides out a local outage and one that shuts itself down for no reason.

Starting up is half the job

When the power comes back, everything has to come up in reverse order, and that is easy to forget. Do the hosts start automatically? Do they wait for the network to be up? Do the virtual machines start in the right order, or do the applications try to reach a database that isn't there yet? Write a start-up plan and check it, not just the shutdown.

Test it – with production's permission

The only time you really know the plan works is when you have tested it. That takes a planned window and production's approval, and it is worth it. A test almost always uncovers something: a network card the UPS doesn't manage, a password that has changed, a virtual machine that doesn't shut down within the time limit.