Here’s the scenario: It’s 3:00 am and there’s a major incident related to one of the most critical services in the IT infrastructure (which is supposed to be operational 24x7 for the customer with a committed availability of 99.98%). The employed monitoring tools have triggered alerts to the service owner and the major incident management team.
This article is about what happens next and how the involved service engineer’s work is regarded.
Introducing the sleeping service engineer
The major incident management team has already opened an emergency bridge, they’ve woken up the service engineer, as per the contact matrix, who was working in the office until 9:00 pm the day before. In doing this, the engineer’s phone rang so loudly that his 2-year-old baby daughter also woke up as did his wife.
Considering the criticality of the service, and the associated business impact, the top company management are also on the bridge and they’re just waiting for the contacted service engineer to join the call (and connect his laptop) before they can start asking why this happened and what caused it (after quickly getting things working again).
Now the restoration story
The service engineer has just managed to put his 2-year-old back to sleep, along with her mom, and is trying to connect to the emergency bridge. The Major Incident Manager and the top management finally hear the service engineer’s voice on the bridge and there’s a collective feeling that Superman himself has come to rescue them. At this point, everyone values the work of the service engineer.
The service engineer does his work. He coordinates with the vendor, updates the major incident management team about the incident status every 15 minutes, answers emails from the top management, and finally the service is back up and running at 5:00 am.
The service engineer – happy that his work is done – then tries to go back to sleep. But he can’t sleep because he is now wondering whether he’s in trouble – despite his middle-of-the-night heroics – because the critical service went down in the first place.
How’s the engineer’s next day?
Despite his lack of sleep, the service engineer is just expected to be in the office at 8:00 am to explain why the service went down and what was done to bring it up.
As soon as he opens his laptop, he sees that he has already had a task opened for him, by the problem management team, to provide the root cause analysis (RCA) details. Plus, there’s a meeting invite to discuss the RCA and the required corrective action needed to avoid future disruptions to the service.
After this there are more follow ups on the action items from the RCA. The service engineer then puts all the necessary controls in place for the future avoidance of service disruption. It has been a really busy, but hopefully personally rewarding, day.
What’s the motivation for the service engineer?
It’s worth thinking about how many organizations really give the right level of credit to the person – the service engineer – who restored the service, as efficiently as possible, and then took the necessary action to prevent it happening again. Rather than potentially just blaming them because the critical service went down in the first place.
Organizations should appreciate that the service engineer’s commitment – from sacrificing their sleep to potentially sacrificing their family commitments for the company. Of course, some will say “Well, it’s their job” or “They get paid well for the inconvenience.” But if it was a great piece of work, then surely the service engineer deserves some form of recognition? The fact that it was a negative situation, or that the service engineer “was just doing their job,” shouldn’t cause people to lose sight of the important role the engineer played last night.
Recognizing the work of great service engineers
Ultimately, the corporate technology is bound to go down or get disturbed once in a while. And someone needs to be there to fix it – even if it’s in the middle of the night.
In my opinion, organizations should make more of an effort to recognize the efforts and work of such service engineers. They should be rewarded with at least a token of appreciation and be recognized in common team meetings or town halls. Service engineers are not machines, they’re human beings and all human beings need motivations for their future actions.
Do you recognize and reward such service engineer successes? If so, please let me know when and how in the comments.