Several years ago, I inherited an IT team that was, to put it mildly, drowning.

Picture this: a team of ten engineers fielding over 1,100 alerts per year. That’s more than three pages every single day and night—a relentless barrage of late-night wake-up calls and weekend interruptions. Morale was in the gutter, burnout was rampant, and it was clear something had to change. When I stepped in, I knew I had to figure out what was broken and how to fix it—fast.

At first, I thought it was a technical problem. Maybe the systems were too fragile, or the monitoring tools were misconfigured. But after digging in, I realized the issue wasn’t in the code or the servers—it was in the team. Not the individuals, mind you- They were awesome. This wasn’t a tech challenge; it was an organizational mess.

The Diagnosis: Silos and Single Points of Failure

The team was split into rigid silos, with seven—yes, seven—separate on-call rotations for just ten people. Some poor souls were essentially on call 24/7, 365 days a year, because they were the only ones in their particular knowledge silo. If a something hiccuped at 3AM, only one person could fix it. If an issue popped up on a holiday, odds were it was the same engineer dragged out of bed. Single points of failure were everywhere, and the team was paying the price. The team was structured for a much larger organization, but didn’t have the staff depth to support it.

The Fix: Cross-Training and “T-Shaped” Engineers

Since budget constraints prevented me from hiring on additional staff, I decided to rethink how the team operated. My goal was to turn them into “T-shaped” engineers—people with deep expertise in their specialty (the vertical part of the T) but also enough broad knowledge (the horizontal bar) to handle Tier-1 support across the board. T-shaped engineering isn’t a new idea, but it seemed like just the right solution for this group. To make this happen, I launched a cross-training program.

We started hosting “Afternoon T” sessions twice a week—think of it as a knowledge-sharing happy hour (minus the drinks). The team took turns teaching each other about their systems, walking through common issues, and documenting everything in a shared knowledge base.

I also intentionally took a back seat and asked the Lead Engineers to coordinate the specifics of the training. They chose their own documentation standards and methodologies. I wanted the team to really own the outcome and drive these sessions themselves, believing that this would further increase the pace and effectiveness of the cross training.

In addition to the Afternoon T sessions, we set up “ride alongs”. Essentially we started buddying up those being trained with the SME for breakfix events. This provided practical, real world experience in addition to the classroom sessions. Over time, the silos started to crumble. The backup guru could now troubleshoot basic VMware problems, and the VMware expert could handle a misbehaving application. They weren’t just specialists anymore—they were a cohesive unit.

At this point, we were ready to start consolidating on-call rotations. Eventually, we got it down to just two: One for IBMi support and another for everything else.

The Results: From 1,100 Calls to Under 20

The impact was staggering. Within two years, we slashed the number of on-call rotations from seven to two. After-hours alerts plummeted from 1,100 a year to under 100. Today? They get fewer than 20 calls per year—a 99% reduction. That’s not just a win for the team’s sanity; it’s a testament to what happens when you take the time to fix the right problem. I’ll admit, seeing the team go from miserable to motivated earned me some serious “boss points”, and it felt good!

The Lesson: IT Isn’t Just About Tech—It’s About People

This experience hammered home a truth I’ve come to live by: IT isn’t just about mastering technology. It’s about understanding people and building teams that work smarter, not harder. The real challenge wasn’t tweaking a server or rewriting a script—it was restructuring how the team functioned.

And here’s a bonus takeaway: cross-training is a real game-changer. Sure, you can make a killing as a hyper-specialized expert, but the most indispensable unicorns I’ve ever worked with are T-shaped. They’ve got depth and breadth, and the utility of that combination makes them an unstoppable human Swiss army knife.

So, the next time you see a miserable team and think you’re staring down a problem, pause and take a step back. It might just be an organizational puzzle waiting to be solved.

And if you find that you’ve got a team drowning in support calls, maybe it’s time to put the cross-training kettle on and start an “Afternoon T” of your own.