How to deliver bad news to your team - Part 1

How to move the cheese

99.9% of the time, a good manager should aim for bottom-up thinking. As a leader, you come up with a set of problems aligned with your company's goals, and your team comes up with creative ways to solve that problem. Let your team self-organize and find solutions. However, there will be moments when you have to force your team to do something they don't want to.

There are two flavors of this. The first one is when you have to re-align your team in a direction you know they will hate, but you, as a leader, firmly believe in the new direction. For e.g, you decide to sunset a project your team loves to work on, but it wasn't contributing positively to the bottom line, so you decide to kill it.

The other flavor is, you have disagreed and committed to an unpopular decision your company is forcing on everyone, and you have to deliver that message to your team. Examples are mandatory RTO, bonuses slashed because of macroeconomic conditions, etc. In this chapter, we will focus on the easier of the two, the former.

First, sit down and organize your thoughts into a structured communication plan. The communication plan should address the following (at the minimum),

1. What is changing?

2. Why is it changing?

3. Why is it changing now?

Here is an example doc that I wrote for myself when I decided to decentralize the on-call rotation and spread it across all product development teams 

1. What is changing?

   1. Starting on mm/dd/yy, the Operations Team will not be the first line responder for paging events generated by our automated system monitors. Instead, the team that owns the specific sub-system will respond directly to pages, 24x7, starting mm/dd/yy

2. Why is this changing?

   1. The Operations Team owns only a specific part of our system, specifically the hosting infrastructure. They have little knowledge about how various customer-facing features work. So when they get paged, they typically turn around and end up calling/texting/paging the product development team's tech lead to fix the issue. As our systems and teams grow, the only logical solution is decentralizing first-line support. Lastly, the Operations Team is not incentivized to fix any long-term sub-system issues that the paging events surface, because they don't own them. The product development teams do.

3. Why is this changing now?

   1. Our application footprint has grown considerably in size and complexity over the last few years. The Production Engineering team gets paged multiple times daily, including nights and weekends. An operationally burdened team with no means to reduce its paging burden will eventually burn out. 

4. How does it affect the daily rhythms of the team?

   1. The Engineering Managers will work with their teams to develop an on-call rotation and runbooks for their services. All teams here at <company_name_redacted> have more than 5 engineers, so assuming a weekly rotation, most engineers will have to go on-call once a month. The engineering managers will be in the escalation chain, i.e., if both the primary and secondary on-call doesn't respond to the page in fifteen minutes, the system will automatically page their managers. I will be the final node in the escalation chain.

5. Can I opt-out of this?

   1. No, everybody in engineering will be part of the on-call process. If you have PTO scheduled on an on-call week, you can swap slots with somebody else on your team.

6. What does success look like?

   1. The goal is to increase the overall stability of our systems. Currently, the team handling the alerts can't fix the underlying issues that caused them because they don't own those systems. As a result, the alerts keep occurring, and the underlying stability issues are not fixed. With this change, the team getting paged can fix the issue causing the alerts, eventually leading to higher overall stability. 

7. Will we prioritize production stability over product development?

   1. In most cases, yes.

Change is hard. Most adults do not like to be told what to do. Resistance to directives is as old as humanity itself. Most people want to experience life in their own way. Stumbling along, failing, learning, adapting, and eventually evolving into better versions of themselves. Product development teams are no different. They want to figure things out on their own. They want to experiment with new technologies, learn from their mistakes and eventually become a better team. So when you, as a leader, give them a directive, expect resistance.

The resistance will come in many forms. The employees will characterize you as an outsider. They will accuse you of being authoritarian. Finally, they will portray you as somebody trying to change the organization's culture. All of those will hurt. 

As you roll out these changes, make sure to make yourself available to your teams to answer any questions or concerns they might have. In those conversations with employees, resist the temptation to revisit decisions. There will always be a group of people who will be passionately opposed to any meaningful change. If you have people-pleasing tendencies like me, you may be tempted to change your decision to please those groups of people. Resist it at all costs. Organizational changes take time to show results. You won't know if you made the right decision in a day or two. It will take weeks and months and sometimes even years. Also, going back on your decision after you have told your team will cause whiplash, and soon your team will start feeling that you are not a decisive leader, further deteriorating their trust in you. Listen to everybody's concerns sincerely, but make it clear to everybody that you intend to stick to the decision you have made for at least six months or longer. Give everybody a voice but not a vote. Once you have made the decision, execute it quickly.

Personal Anecdote

In a galaxy far away, I inherited an engineering team that was growing quickly but experiencing significant teething pains. The team that was struggling the most was the Production Engineering team that managed the hosting infrastructure.

In my second week at the company, I got a very unusual request from one of the lead engineers in the Production Engineering team. He wanted an Executive Assistant to help him out. I stared at the email for a while to ensure I understood the request correctly. Executive Assistants are personal assistants who assist senior leaders with time management, travel, events, etc. They are typically attached to only executives (C-Suite, VPs etc.), so I was stunned at this request. I was new to the organization, so I didn't want to scoff and deny the request immediately. Instead, I decided to find out more. He replied with a backlog of items he was drowning in that he thought an EA could help. Items ranged from pending requests from the developers to backlogged requests from the finance team about cost management. All those things seemed legitimate, but what was puzzling to me was why he and his team could not handle that load. The load was definitely on the higher side, but it seemed doable for a team of eight, so I dug in more. After some quick digging, I found the wrench jamming up the team's gears. The root cause was the on-call load. This Production Engineering Team, for some unknown reason, had decided to act as first-level support for the entirety of the engineering organization. This team of eight was responding to any system incident across the application regardless of whether they ever worked (or owned) in that part of the system. When I asked them why the rest of the engineering team was not stepping in to offset their load, the answer I got was the rest of the engineering team was not comfortable going on-call because they had never done it before. 

Being on-call can be stressful. You could get paged in the middle of the night and be expected to fix a system issue affecting a large swath of your customer base. It requires the engineer to think on their feet and make quick decisions with very little data. When the application is down, there isn't much time to spare. Sometimes your decisions might prove incorrect, so the engineer also needs to be comfortable pivoting quickly. I understood why the development team was reluctant to take on the on-call responsibility. However, the pressure cooker situation the Production Engineering team was in was untenable. I had to make a change quickly before that team decided to rage quit together.

I decided to distribute the on-call load across all teams to offset the load from the Production Engineering team. That was going to be tough. Nobody likes it when their cheese moves, especially if it moves to a completely new maze, set up in a completely different room watched over by a strange new creature, a.k.a; the new engineering guy. 

I first broached this change with the people leaders reporting to me. As I talked to them, it became clear that the biggest fear in their teams was the fear of the unknown. 'What do I do when I get paged?', 'What happens if I sleep through the page?' 'What do I do if I need to pull in some other team?' And the biggest fear of all, 'What do I do if I don't know how to fix the problem?' 

Once I got a sense of what the engineers were afraid of, I put together a rollout plan which addressed most of their concerns. I worked with the Production Engineering team to develop a training course I put all teams through. Then I picked a couple of moderately enthusiastic teams about this change and empowered them to lead the charge by coming up with an on-call rotation, runbooks, and practice drills. Additionally, I encouraged them to share their learnings with other teams. However, the most important thing that the engineers cared about was the safety net. Who is going to help if I get stuck? Am I doing this the right way? To put my team at ease, I added myself to the on-call rotation for all alerts. This means I get paged if anyone on my team (except the Production Engineering team, because they already had a rotation going) got paged for anything. I did this for the next three months. Most of the time, I only acted as a rubber duck for the engineers who were oncall. For people who don't know, the act of rubber ducking is to narrate your thought process as you solve a problem to someone else who isn't going to respond with any opinions of their own. Just narrating the problem to someone else will reveal the answer to you. 

I did this for about three months, and at the end of it, the team was comfortable enough to start handling on-call rotations by themselves. Just showing up alongside my team and supporting them, significantly increased my credibility. As Patton says, "Do Everything You Ask of Those You Command." Also, it helped me get a crash course on the system's intricacies, including the friendly gremlins who live in them. 

In about six months, on-call rotation became a way of life for my team. Another successful transformation was in the bag.

A quick recap of takeaways from this chapter

1. Remember that in some situations, everybody gets a voice, not a vote.

2. Before communicating changes to your team, write down,

   1. What is changing?

   2. Why is it changing?

   3. Why is it changing now?

3. Expect and plan for resistance from your team

4. After you decide, execute it quickly

5. Expect to lose a few people