Situation
A food manufacturer in Marijampolė, Lithuania, with about 250 employees, works two shifts and supplies retail chains with strict delivery windows. The IT infrastructure is typical for manufacturing: several virtualised servers running the ERP and warehouse systems, separate production line control servers on the production network, a few dozen workstations, network equipment with VLAN segments between the office and production, and Microsoft 365 for e-mail and documents.
Backups did exist, but only for the servers: Veeam made virtual machine backups to local storage in the same server room. The line control servers were not backed up, because they were maintained by the equipment vendor and nobody was sure it was allowed. Network device configurations were stored nowhere except on the devices themselves. Microsoft 365 was regarded as “Microsoft’s concern”. Recovery had never been tested.
There were two reasons to get in touch. First, an audit by a retail chain asked for proof that the company can resume operations after an incident, and “we have backups” was not a sufficient answer. Second, when a single switch failed, the production network stood still for half a shift while its configuration was reconstructed from memory. The production director asked a simple question: if the server room were gone tomorrow, how long would it take us to produce again?
What was needed
- Backups for the entire infrastructure: servers, line control servers, workstations, network device configurations, Microsoft 365.
- At least one copy outside the factory and unreachable from the company network.
- An agreed and tested recovery time for critical systems, which production can be planned around.
- A recovery location in case the company’s own equipment were lost.
- Documented drills, with a report that can be given to auditors.
What we did
Weeks 1-2: inventory and criticality list
We listed everything that has an IP address, and together with the heads of production, logistics and finance we determined for each system how long the business can work without it. Three came out as critical: the ERP, the warehouse system and the line control servers. For these we agreed on a 4-hour recovery target, and for the rest, by the next business day. This list later became the drill scenario.
Weeks 3-6: backups for the whole infrastructure
We restructured the server backups following the 3-2-1 principle: Veeam backups on site, a second copy in the MDP CLOUD data centre in Lithuania, and a third copy that is immutable and invisible from the network, which ransomware could not reach even with administrator rights. With the line control equipment vendor we agreed that their servers are backed up with writes to the database paused during a nightly break. This took the longest, because it needed the vendor’s written confirmation.
We began collecting network device configurations automatically every night into the same backup repository, with a change history. For workstations running specialised software (weighing scales, label printing) we created image backups. We placed Microsoft 365 e-mail, OneDrive and SharePoint into separate Microsoft 365 backup, because Microsoft retains deleted data only for a limited time and does not cover every scenario.
Weeks 7-8: PlanB
For the critical systems we prepared PlanB: a pre-configured environment in the MDP CLOUD cloud into which the ERP and the warehouse system can be brought up from backups without buying any hardware. The environment waits switched off until it is needed; once switched on, employees connect through VPN from any location.
Weeks 9-10: drills
The first drill took place over a weekend: we simulated the loss of the server room and restored three critical systems in the PlanB environment. The first attempt took slightly more than 4 hours, because the ERP licence file turned out to be tied to the old server name. We corrected the procedure and repeated the drill two weeks later, and this time we met the target. Separately, we restored the configuration of one switch onto a spare device. It took about 20 minutes, not half a shift.
Result
Recovery of the critical systems has been tested at 4 hours, which is no longer a target but a fact recorded in a drill report. We gave the retail chain’s auditors the drill report, the backup policy and the recovery procedures, and no questions were left.
All network device configurations, 100%, are stored with a daily history. A change someone made on a switch on Tuesday is visible on Wednesday and can be rolled back if needed.
Recovery drills take place twice a year on an agreed schedule, each time with a different scenario: loss of servers, a ransomware attack, deletion of a Microsoft 365 account. The report of each drill goes to management and is kept for audits.
“For the first time we have an answer to the question ‘how long would it take us to produce again’, and it comes from a report, not from someone’s head.”
What we would advise in a similar situation
- Backups without a tested recovery are an assumption, not protection. The first test almost always reveals something you did not know.
- Include everything that has a configuration in the inventory, not just servers: switches, firewalls, scales, printers.
- Agree on backups of production equipment with the equipment vendor in writing, before an incident happens.
- Set the recovery target with the heads of production and finance, not in the IT department: they know what an hour of downtime costs.
