Situation
A manufacturing company in Kaunas, Lithuania, with about 120 people, had a single IT employee for more than ten years. He knew everything: where the servers stood, what the passwords were, how the production accounting program worked, which printer on the shop floor jammed and what to do then. There was practically no documentation. It was in one person’s head and in a few personal note files.
The employee left after giving two weeks’ notice. In those two weeks the company realised that there was nobody to hand over to and nothing to hand over: finding a new IT specialist takes a manufacturing company months, and people with that profile are scarce on the market. Meanwhile daily operations began to stall. Employees wrote to the finance manager about email that was not working, the warehouse manager tried to reboot a server himself, and nobody could say whether backups were being made at all.
The company’s manager came to us with a specific question: is it possible to take over IT support in a way that keeps production running and makes the company no longer dependent on a single person?
What was needed
At the first meeting with the manager and the finance manager we defined what was essential and what could wait:
- A helpdesk for employees from day one, with a clear response time, so that requests would no longer go through the finance department.
- Take over two physical servers (production accounting and a file server) and about 90 workstations, including the computers on the shop floor.
- Find out what backups were being made and make sure they could be restored.
- Review access: the former employee had administrator rights everywhere, and some of them were shared.
- Document the entire infrastructure so that the next time people change the company is not in this situation again.
- Budget: no more than the cost of the former IT position including taxes.
One additional constraint: production runs two shifts, so any work on the servers was possible only on weekends or at night.
What we did
Week 1: takeover and inventory
On the first day we turned on the helpdesk: employees received one email address and one phone number, requests are logged in a system, and the response is within 30 minutes. At the same time our engineer carried out an infrastructure inventory: servers, network equipment, workstations, licences, applications and suppliers. The result was the first description of the infrastructure the company had ever had.
The most important finding: backups were being made to an external disk attached to the same server, and the last successful backup was seven weeks old. Nobody had ever tried a restore.
Weeks 2-3: access and backups
We changed all administrator passwords, removed the shared accounts and introduced a separate account for every employee. We turned on two-factor authentication for server administration. The former employee’s access was disabled at every supplier, including the domain registrar and the internet provider, which is often forgotten.
We moved backups to a server backup solution with Veeam: daily backups to our data centre in Lithuania (EU) and weekly backups to an Amazon data centre in Germany. We tested recovery in the third week: the production accounting server was restored in a test environment in just under three hours.
Weeks 4-5: workstations and servers
On all workstations we deployed a monitoring and remote management agent, fixed the updates (some computers on the shop floor had not been updated for more than a year) and standardised antivirus protection with Bitdefender GravityZone. For the servers we introduced a server management regime: round-the-clock monitoring, with updates in the night from Saturday to Sunday, agreed with the production director.
The hardest part was the production accounting program, which was supported by a small local vendor. It turned out that its database had no maintenance at all, and its logs took up most of the disk. We fixed this together with the program’s vendor, and the database is now included in monitoring.
Week 6: documentation and SLA
In the last week we completed the documentation: network diagram, a list of servers and services, an access register, recovery procedures and supplier contacts. The documentation is kept by us and by the client, and updated with every change. We signed a service level agreement with specific response and resolution times by request severity.
Result
In the first three months after the takeover the number of IT requests fell by about 35% compared with the first month. The reason is simple: most recurring problems (computers that were not updated, printer drivers, full disks) were removed systematically rather than being fixed from scratch each time.
Helpdesk response under the SLA is 30 minutes, and it is measured for every request: each month the client receives a report with the number of requests and the response and resolution times. The finance manager no longer gets IT calls.
The infrastructure is 100% documented: every server, service and access has a record. Backups are checked daily, and recovery is tested every quarter. The monthly cost to the company is lower than the former IT position.
Before, IT was one person whose holidays we dreaded. Now it is a service with a report and a contract.
What we would advise in a similar situation
- Do not wait for your IT employee to leave. Documentation and an access register need to exist now, even if the person has no plans to go. This is not distrust, it is a company asset.
- The first job when the person responsible for IT changes is to change all administrator passwords and disable access at suppliers, not just inside the company.
- Check that your backups can be restored. A backup that nobody has tried to restore is an assumption, not protection.
- When choosing an IT support provider, require a written SLA and monthly reports. Otherwise you will have no way to judge whether the service works.
