The Learning Happens in the Debrief

Applying a key tool of a fighter pilot's success to identify and learn from our Covid missteps

Note: I wrote this post before The Atlantic published Professor Emily Oster’s piece: Let’s Declare a Pandemic Amnesty. While this was not intended to be a response, I think she is wrong—not necessarily regarding tone, but certainly on whether we need a painstakingly detailed “look under the hood” for every decision at every level made by everyone who had a hand on a lever that affected others, from the daycare down the street to the CDC and White House. We do. A “pandemic amnesty” doesn’t help us get there. A “Covid debrief” would.

As we approach our third Covid winter, the “do something” chorus has grown noticeably smaller, though the shrinking number who remain in that chorus are among the loudest and most devoted voices. However, many former devotees have finally stumbled upon what seemed plain to so many of us: “doing something” has costs, and those costs have often exceeded whatever benefit the decisionmakers hoped to achieve.

We have finally removed the tag of “misinformation peddler” from those who want to dig a bit deeper into and opine on certain topics, including:

• Many interventions failed to pass a cost-benefit analysis (mainly by lacking in benefit), including prolonged school closures/online learning, Covid vaccine mandates, and mask mandates

• Politicos, the media, and the public health apparatus consistently oversold both pharmaceutical and non-pharmaceutical interventions

• Vaccine recommendations in the US, which act as quasi-requirements (for those who don’t want to be booted from their college, for example), have varied significantly from other western nations

• Origins of Covid-19

Some people still maintain that our only missteps involved too few interventions and that the US should have done even more. Even now, several non-troll Twitter users—most in the public health space—clamor for their return. Part of the problem with the current discourse is that in most Covid discussions or debates people tend to wield a broadsword when a scalpel is needed.

The Fighter Debrief

A good fighter debrief would serve us well today for Covid. I’ve been in many such debriefs, and I will tell you that they can be absolutely miserable. One of the more painful ones that I recall was around 6-7 hours. This was for a 1-hour training flight as part of an instructor upgrade for my old flight commander. I knew we were in trouble when the evaluator asked, “Is anyone on the flying schedule tomorrow?” Nobody was, which meant that mandatory crew rest wasn’t an issue. Shit.

Allow me to paint the picture of a fighter debrief: we break a flight down into the following components: plan, products, brief, admin, tactical admin, and execution. We will debrief each as long as is necessary to identify mistakes so that we get closer to perfect execution (which is unattainable, of course, because something could have been done better or cleaner). Was the plan good or not? What did we miss? Who was responsible for which product? The target set? Images? Was the flight briefed correctly? Was there a question mark hanging over your head as you stepped to the jet? It’s all coming out.

The most important (and thus the longest portion) is the execution. What did each person do in the air and why did you do it? Here’s what an excerpt might sound like (I’ve tried to “civilianize” the terminology for you):

• Instructor: “Play [the cockpit playback video of our screens]. [5 seconds later]. Stop! Lead, what the fuck? Where’s your wingman right now?”

• Lead: “I think he’s...”

• Instructor: “I’m not asking you to think. Where was your wingman at this point?”

• Lead: “8 o’clock at 1.5 miles. 25K feet.”

• Instructor: “That’s right, lagging—and low. Wingman: why the hell are you still at 25K feet?”

• Wingman: “I don’t know.”

• Instructor: “Did we discuss in the brief about when we need to climb to commit?”

• Wingman: “Yes.”

• Instructor: “And did you just fire a missile here—at this range and at your altitude?”

• Wingman: “Yes.”

• Instructor: “Why?”

• Wingman: “I was hoping to take out the enemy aircraft I had locked.”

• Instructor: “Is that missile—at your range and altitude—going to do anything to help in this scenario?”

• Wingman: “I’m not sure.”

• Instructor: “You’re not sure. But you went ahead and fired.”

• Wingman: “Yes.”

• Instructor: “That’s unsat, Wingman. Completely unsat. Ok, let’s keep going. Play. [10 seconds later]. Stop! [Proceeds to lay into someone else for messing something up].”

Believe me when I tell you that this understates the intensity of many upgrade debriefs. The bottom line is that we deconstruct everything from the movement of the throttle to each actuation of our controls to the degree of depth necessary to learn and fix our mistakes. “What did you do here? Why? Did you have to flex because the plan changed? Did you consider option X? What about Y? Z? Any other options? Why did you choose Option M over Z?” It’s all fair game, because we have to get better so that we can get it right if and when lives are on the line.

When you’re in the hot seat, you feel like utter shit as you’re getting peppered. So what. I’ve been there many times. We all have. It doesn’t matter. This is our profession (cue the Hyman Roth quote from Godfather II). We were harder on ourselves, in fact. Folks hang around the vault practicing what they messed up, going through the tactics papers, and talking with others to try and improve their decisions next time. And by the way, sometimes the answer is in fact gray—a choice among two or three equally good (or bad) options. But the discussion centers on specific reasons for specific decisions, otherwise there is no learning and no improvement.

Covid Debrief

I don’t think we can cram 10,000 different decisionmakers in a fighter debrief room for a single Covid debrief to “play back the tape,” but that doesn’t mean we can’t apply similar principles. We need to touch with a needle each decision, including the specific prompt or data or event that drove a specific decision at that specific time (or, conversely, the delay in removing a restriction). We need to find out exactly what second- or third-order effects were considered at each decision point, if any, including how and by whom they were considered. Decisionmakers absolutely will (and should) feel like they’re in the hot seat.

It is unacceptable for a pilot to say, “Well, there were bad guys out there trying to kill us so I just dropped my bomb or lobbed my missile.” Jaws would hit the floor in the debrief. It is equally unacceptable for a city council spokesperson, or public health official, or governor to say, “Covid was killing people and it was scary as hell so we took a bunch of actions to be safe.” It may be the truth—and probably is, in most cases—but it’s still unacceptable. Specifics matter. Something like this in a Covid debrief would be nice:

• “You instituted a statewide order on [date] stating that [content], and that order remained in effect until [date]. Let’s walk through each piece of that order, and I want to talk about all the data and stakeholder input that went into each of these specific provisions, including why you made them at this time and any external advisers on whom you relied.”

Worse, without some version of a debrief, we don’t know who still thinks every decision they made was perfectly correct and justified by the situation on the ground and the information available at the time. Who currently has their hand on a remote-only school lever and would still happily pull it in December 2022? Anyone? Bivalent vaccine requirement for otherwise vaccinated 13-year-olds who have had Covid twice? Anyone in that camp? Who? Based on what specific data? People would like to know. Debriefs can reveal this information as well. You think your approach was right? Ok. Why (and be specific)?

If you want to read a bit more about fighter debriefs, go read Hasard Lee’s Taking the Blame: Why Fighter Pilots Have to Own Their Mistakes. For those who don’t have the time, consider the quote below:

Taking ownership is a skill. As fighter pilots, most of us are predisposed to win at all costs—within the rules and regulations. In the debrief though, with the mission already flown, the way to win is to accurately identify lessons that will make everyone better for the next flight. It’s a fragile environment that only works when everyone is willing to first look inward for failures to the mission.

Now ask yourself whether this describes our politicians and other decisionmakers. I have a good guess as to your answer. Many reading this, like myself, understand that CYA, chasing favorable media, and appeasing the important voices on your “team” (which together = votes) usually win the day and that looking inward is rare.

Accountability is the only real tool that the public has. The political will to hold oneself accountable and take the heat in a genuine Covid debrief is non-existent. So others must hold those making the decisions accountable—repeatedly and with specificity—until the issues are no longer relevant. Because we are not there yet, the “mistakes were made” amnesty route does not work.

---