Today, I had two meetings. One of them was with an old client for planning some future interventions. It was calm and positive, so overall, a peaceful meeting. The other one, as I expected, was less so.

I've been working with this client since 2012. When I first met them, they had a physical server (running Windows 2003 Server) and wanted to upgrade to Windows 2008r2. The person managing it internally was skilled in handling Windows but had no background in system administration or virtualization. Since they were close to retirement, they preferred to request consultancy.

We designed a migration project to move the Windows 2003 server from physical to virtual, created a new Windows 2008r2 VM, and set up a backup procedure. The owners were happy, and the internal staff was satisfied. The only critical issue: a piece of software developed by a very elderly programmer, on which the entire operation depended.

The company grew, eventually reaching eight locations spread over a fairly large area. Each location had a main virtualization server and a backup server ready to take over. It wasn’t a true HA setup, but recovery times were quick and predictable. In the most critical locations, there was a cluster of three machines (based on Proxmox, Ceph, in HA). In a couple of cases, they restarted instantly without even noticing failures. All clients effectively became thin clients, connecting via RDP to a Windows server for daily use. Not ideal, but at least they got rid of PC management, which had become difficult given their rapid growth.

When the internal IT manager retired, they considered appointing a successor. In the end, the external developer took over everything, remaining as a consultant and stalling as much as possible. In general, the owners (wisely) always chose to have various people collaborate rather than impose strict hierarchies. The problem was that hierarchies sometimes emerged based on gerontocratic presumptions. I'm used to these situations and have always adapted, smoothing out problems. I had a few heated exchanges, but in the end, everything kept moving forward.

Until a few years ago, when COVID started changing work dynamics.

Suddenly, I was only consulted for problems. I sent concerned emails proposing updates and maintenance, but I never received responses. Or, if I did, they came from this person - or other external figures whose roles I didn’t understand - with promises and reassurances, but never direct solutions. However, whenever problems arose, they would immediately reach out - but again, never the (new) owners or their close collaborators. This became increasingly frustrating.

At some point, a new figure appeared - someone unknown to me. A consultant who proposed "miraculous" (but expensive) solutions. Among them, the idea of centralizing all servers and services in a single location (connectivity: 50 Mbps inbound, 20 Mbps outbound). When I found out (through indirect sources), I nearly had a heart attack. Their connection was unstable and slow, their services were critical, and downtime was a major issue. The key to their stability for many years had been managing everything locally with failover solutions, and now they wanted to centralize everything? I imagined over 160 people connecting via RDP to a single virtualized Windows server through that connection. Crazy. But they insisted: "this is the future of the infrastructure, here. The owners have decided."

I called the person in charge, expressed my concerns, and sent an email to the owners. No response from them. Meanwhile, the responsible person simply said: "This is the decision, and this is how it will be done. We just ask you to keep the current setup running for about a year."

Since I can’t impose rules in someone else's "house", I agreed to keep my setup running for at most one year. I asked for permission, at least, to perform system updates. No luck - all the budget was allocated to the centralization project.

I let it go and only intervened when issues were reported - fortunately, rarely.

Years passed. Two, three, four. A few calls, a few reports - then suddenly, serious problems every day. Servers shut down, cables unplugged, etc. Out of nowhere, they informed me that the person I had been dealing with (the responsible person) had stopped working for them. They appointed a new person in charge (who had already been collaborating with the previous one). This new person started calling me at all hours, for anything and everything.

Fine, but - I asked - what happened to the centralization project?

"Still in progress, the budget is high, and we need more time."

What? Years? I didn’t get it. But I moved on. Meanwhile, the usual problems: cables unplugged, servers being shut down, VMs being powered off (and very few people had the admin passwords), strange situations. I’ll omit many of them, especially the most critical or suspicious ones. I posted about some of these in the past, so I won’t repeat them here.

One evening, I started receiving notifications of failed backups. The next night, the same. This continued for days, so I logged in to check (even though I was nauseated by this whole thing - if I make a commitment, I follow through). Someone (I can guess who, considering only one person had the credentials to do it) broke a VM, restored it from backup but placed it on the wrong storage - the backup storage - saturating it and breaking everything. VM and backups.

This was the second time in two weeks that this kind of operation had been performed.

I called this person. He politely replied that he was in charge and did whatever he wanted. I pointed out that the owners should be informed that something was wrong and that intervention was needed because the infrastructure was in a critical state.

"Don't worry, I’ll talk to them. I’ll let you know."

I never heard from him again. I tried calling back.

"I spoke to them. For now, we’re not doing anything. When something breaks, I'll call you, and you fix it."

I lost it. I sent an email requesting a meeting with the owners. No response. I had no direct contacts, except for someone they deeply trusted - but that I didn't want to disturb.

I finally spoke with this person. Either they let me talk to the owners, or I would stop responding and intervening. I had no contracts or agreements (I proposed one in 2020, but, as for all the other communications, it was ignored).

I was out of my mind. This person tried to calm me down, but they couldn’t. I stood firm. I would not take responsibility (even moral responsibility) for others' mistakes. Either we clarified things, or I was out. Panic. It became obvious that they couldn't deal with any problems without my help.

A meeting was scheduled - then postponed by a few hours. But before that, they had a meeting among themselves.

The current owners recently took over direct management and had never dealt with these matters before. I was told they were unreachable, inaccessible. That already seemed suspicious to me, but...

I entered the meeting, ready for battle.

The meeting started. I needed to understand whether the owners had been ignoring me or if, for some reason, they didn’t know what was happening.

During the meeting, one of those people decided to come clean, scared of how the owners might react.

Surprise (but not really): the owners were aware of my role but not of everything else. My emails had never reached them. My communications were always filtered by the people between me and them - people who took credit for my work ("I restarted the server," when in fact, I did) and carefully ensured that none of my ideas or proposals reached the owners.

My hypothesis: their goal was not the company's well-being but their own self-interest. Expensive for the company, profitable for them.

The meeting had been rescheduled specifically to keep most of those people out, knowing their schedules. And, in fact, they weren’t there.

Long story short, when this trusted contact reached out to the owners, they started investigating and uncovered quite a few things - including the deliberate filtering of my communications to exclude me. They just allowed me to work and bill a little but knew I would expose too many things.

And they couldn't allow that.

The cost savings of my solutions (compared to the tens of thousands already spent on the still-unfinished "centralization" project) would have raised serious questions among the owners. And they were already irritated by the waste of money.

The Result?

- The IT infrastructure will be redesigned (based on FreeBSD).
- Redundant servers (ZFS replication) will be deployed.
- New setups for other services.
- The people responsible for the mess? Fired
- The owners are considering legal action against one individual for tampering with documentation and communications.

I solve problems.