Patching without stopping production

In an office, patching is a scheduling question. You pick a night, the servers reboot, and in the morning everything is back to normal. A factory doesn't work like that. The servers in the factory's datacentre run systems that control, monitor and trace production, and a reboot at the wrong moment can mean a line standing still, a batch that has to be scrapped or a delivery date that slips.

At the same time, not patching is not an option. Unpatched hypervisors, old firmware and backup software several versions behind are exactly what make an OT environment vulnerable. So the job is not to choose between security and production. It is to plan so that you get both.

Decide which versions you move to – and which you skip

The first decision is not when to patch, but what. In an IT environment with internet access, staying close to the latest release makes sense. In a tightly segmented factory network the calculation is different: every upgrade is a window of risk, and the network's isolation gives a protection that means you don't always have to be first.

So I work with an explicit version strategy for the hypervisor, vCenter and backup. It answers three questions:

• Do we need what the new version brings? A security fix for an exposed component weighs heavily. A new feature we don't use weighs little.
• Do the versions fit together? Hypervisor, vCenter, backup software and server firmware all have compatibility matrices. Upgrading one part without checking the others is the most common way to create a new problem while fixing an old one.
• Who else is affected? If backups are replicated to another datacentre that someone else owns, the versions have to line up with their plans too.

Fewer, well-chosen upgrades beat many small ones. Every version you skip is a window of risk that never opened.

Production owns the maintenance window

In a factory, production decides when something may be switched off. That sounds obvious, but it shapes the whole plan:

• Windows are often at weekends or during planned maintenance stops, and they move when the order book changes. The plan has to survive the date changing at short notice.
• Asking the site's IT contact is not enough. You need someone who understands which systems are actually affected when a given server reboots, and what that means on the shop floor.
• When several factories in different countries are upgraded, time zones, local holidays and language become part of the project, not a footnote.

The hardest part is rarely the technology. It is the coordination: with production, with the people who own the network outside the factory, and with the colleagues doing the work on site or remotely.

Let the cluster carry the load

The most important technical condition for patching without downtime is an architecture that allows it. A virtualisation cluster with enough headroom can take one host at a time out of service, move the virtual machines with vMotion, patch the host and put it back, without the applications noticing.

That only works if you have done the maths beforehand. Can the cluster run full load with one node gone? Are there storage policies that need a certain number of nodes for data to stay protected? In a small three-node cluster the margin is thin, and it is better to know that before the maintenance window than during it.

Anything that can't be moved, such as standalone physical servers or systems licensed to specific hardware, needs its own plan and usually a real stop. Find those early.

Firmware is patching too

It is easy to focus on the operating system and hypervisor and forget the server's own software: BIOS, network cards, storage controllers and the remote management card. Vendors publish bundled packages with tested combinations, and those are what you should use, not individual updates someone happened to find.

In isolated networks with no internet access, the packages don't arrive on their own. They have to be distributed in a controlled way, with checksums and traceability, through the same secure access path as everything else. Preparing that takes time, but that isolation is exactly what makes the environment worth protecting.

A rollback plan you can actually carry out

"We'll roll back if it goes wrong" is not a plan. A plan answers:

• What is the checkpoint where we decide? For example: if the host isn't back in the cluster within a set time.
• How exactly do we restore: snapshot, backup, a reinstall with saved configuration?
• Has the backup of vCenter and critical systems been taken before the window, and has it been checked?
• Who makes the call, and who tells production?

Write it down, for next time

Every upgrade teaches you something. Which host took longest, which step in the instructions was unclear, which check was missing? If it isn't written down, the next person makes the same mistake. I write a short review after each campaign and update the instructions, so the next round is a little calmer than the last.