Tampering and routine variation

What do you do when a new data point drops—and all eyes turn to you? In this episode, John Dues and Andrew Stotz explore the leadership discipline required when performance data changes. Instead of reacting to a single data point, they unpack how Deming thinking (understanding variation, avoiding tampering, and pausing to interpret patterns) can protect trust, stability, and improvement. A practical conversation for leaders who want wisdom—not speed—to guide their decisions.

TRANSCRIPT

0:00:02.3 Andrew Stotz: My name is Andrew Stotz, and I'll be your host as we dive deeper into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is when the numbers change and everyone looks at you. John, take it away.

0:00:28.4 John Dues: Yeah, it's good to be back, Andrew. I think this is sort of an interesting topic. Many of us that have been in leadership roles have been in this position where the numbers change, whatever they may be. For me, they're dips in attendance, they're assessment results changing, something like that, a subgroup's results changes from the previous year. Sometimes the changes are small, sometimes they're big. But I'm thinking about times when they're just large enough to draw attention in a meeting. And it's not even really so much the size of the change that's important, it's what happens next.

0:01:12.9 John Dues: So you can kind of put yourself in one of these meetings where you're looking at data and maybe you didn't even expect it, but people kind of notice. Then someone asks what went wrong? And then the next thing that comes is someone suggests some type of fix or solution, and then this pressure starts to build. Especially if they're all sort of looking at you, the silence can feel irresponsible. And so what do we do? We react in some way. We call another... For explanations, maybe from others. We adjust a plan that's already in place. We launch a new initiative or tighten expectations on people, whatever it may be. None of it's out of malice. It's done out of care, most typically, or at least in the settings I've observed this sort of phenomenon.

0:02:13.1 Andrew Stotz: Don't just stand there, do something.

0:02:15.2 John Dues: Don't just stand there, do something. But the thing is, very often it just makes things worse. Right?

0:02:21.0 Andrew Stotz: Don't just do something. Stand there.

0:02:23.8 John Dues: Right, right. The opposite. But even if you know that, it's very, very difficult in the moment to...

0:02:32.5 Andrew Stotz: The pressures.

0:02:33.6 John Dues: Yeah.

0:02:34.9 Andrew Stotz: Well, I have a little... Little thing happened last night when a friend of mine came to see my mom and me, and we went out for there's a restaurant nearby, so we got the walker and got mom going. And her natural inclination was to help mom in getting up and that type of thing. And I was explaining to her the difference between what I call a caregiver and a caretaker. And I was saying that most people are caretakers where they're just taking care and they want to just help. And she's like, "It's irresistible. I mean, in my bones, I want to help." And I said, "It's very hard to see that sometimes the best help is to let her struggle and use her legs to get up, not to help her on that." And that was like a revelation for her last night, it just made me think about that.

0:03:33.8 John Dues: No, that's actually a perfect analogy because her health is sort of a high stakes environment. Just like schools are high stakes environments or many of the businesses that people run that listen to this podcast have high stakes. In our cases, it's students and families matter, outcomes matter. There's a lot of different stakeholders that are interested in what's going on in schools. And when those numbers do change, it can feel like neglect if you don't do anything. We're expected to notice. We're expected to... Good leaders are supposed to respond. They're supposed to act decisively, right?

0:04:12.0 Andrew Stotz: Yeah, because there's another aspect to it too. Let's just say that you have a boss that understands it and you're like, "Yeah, it's just noise. It's not signal." But how many times can you say that? Right?

0:04:27.8 John Dues: Yeah, that's right.

0:04:28.5 Andrew Stotz: That's another kind of pressure in that situation.

0:04:31.6 John Dues: Yeah, that's like the second-in-command type person, right? So they have their own pressure. And what you can see happening, this like visible action is sort of like evidence of competence because you can see it. And so the reaction becomes the default. So just like in this example you're giving with your mom, that action to help is very hard to resist. Even though by doing so, like you were saying, she doesn't get the physical exercise and actually makes things worse in the long term for your mom's health.

0:05:10.4 Andrew Stotz: "Boss, why did Kevin get a promotion and not me?" "Well, Kevin's a man of action."

0:05:14.8 John Dues: Man of action, right. Exactly. Exactly. And there's all these risks for a leader that doesn't react right away. Are they disengaged? If they're asking questions instead of acting right away, are they just uncertain? They lack certainty? Are you ignoring the data if you are pausing or waiting? Again, under these conditions, which I think are prevalent just about everywhere that I've been, at least, reacting quickly feels like the safest move. But I think the conflation is speed and wisdom. But speed is not, definitely not the same thing as wisdom, right?

0:06:02.1 John Dues: In all of our organizations, the data fluctuates naturally over time. No different in schools, like we've talked about. Attendance rises and falls, assessment results bounce up and down, behavior incidents they spike and they dip. And it's not necessarily a sign that something's broken. It's often just how systems typically behave, the systems that we're paying attention to. I think the main mistake leaders typically make in that moment when they see that movement is that they think that automatically means something changed. And so you get these concerns if it's a bad move in the data. If it's a short-term increase, maybe we trigger some type of celebration. So this works both ways, actually. But the main point is that one data point becomes a story. It becomes the story of what's... We try to attach an explanation to this dip or this increase that's actually not grounded in any kind of reality. We would say they're just reacting to noise, kind of like what you just said. And the problem, though, is that there's a number of then very predictable things that happen. First, educators, and I felt this as a teacher. I taught in Atlanta Public Schools, a big district that was trying lots of new things in the early 2000s. You feel this whiplash. So priority shifts, guidance changes. Yesterday's focus is replaced by today's concern.

0:07:44.5 John Dues: And what happens in a setting like that, that I found, is that people start explaining instead of learning. Especially when there's a strong accountability system like there is in education systems, results are questioned immediately, often. And so the safest response at almost all levels of the organization is just to justify what's already happened, not to explore what might be improved. Very, very, very difficult. And that then leads to trust eroding. And over time, what I've seen is that educators learn that any fluctuation brings scrutiny. They become cautious, defensive, quiet. And obviously none of that improves outcomes. And again, just like in the example with your mom, it actually makes things harder to improve in the long term. So this overreacting to this routine variation then often increases variation, and so the system actually becomes noisier and not more capable. You get this vicious cycle. What's that?

0:09:00.5 Andrew Stotz: Tampering.

0:09:01.8 John Dues: Yeah, tampering. Exactly. That's what Deming would call it, tampering. When you intervene in a stable system.

0:09:07.3 Andrew Stotz: It's interesting. The one data point becomes a story is a great, great line. In the world of finance, everybody's trying to get the next wave. As a financial analyst, you're trying to think, okay. And all we do constantly is look at the next data point and say, "Does this confirm or not my view that gold's going to crash now, or gold's going to rise, or US stocks are going to X, or the dollar is going to... " And most of the time, we're just making one data point become a story, and then the next data point comes out and it's like, "Okay, so there's a different story here." And then...

0:09:51.3 John Dues: Yeah. That explanation there it's sort of... The key idea is reaction. It's literally seductive. It is seductive because it feels productive.

0:10:04.3 Andrew Stotz: In my finance work, when I help people with their money, what I do introduce what I've learned from Dr. Deming to say it really helps me separate the signal from the noise in the stock market, and therefore, I will never react. And I even set parameters where I rebalance my portfolio every three months. So when they go, "What are you going to do about such and such?" it's like, "Everything's set. I'm going to wait until the results are in, and I'm going to reevaluate on a framework, on a systematic way," which just helps me from getting whipsawed this way or whiplash this way or that way. And it's proven to be not only great for helping people feel like I have a deeper understanding and follow what I'm doing, but it also improves performance.

0:11:07.7 John Dues: Yeah. And you know, I'm definitely no financial expert by any means, but it makes me think of The Big Short, the movie, when I don't know how true to reality it is, but when the character played by Christian Bale, Michael Burry, is sticking to his guns with his shorting of the housing market and people are coming into his office and screaming at him. He's getting emails that are coming in one after another calling him an idiot, threatening him with lawsuits, and he holds. So that's like an extreme example of not reacting to noise. And you can see what it does to him in the movie, the intestinal fortitude, before sort of it comes to the conclusion. He got less and less certain even though he stuck to his guns, that he was doing the right thing. Right.

0:12:00.3 Andrew Stotz: I got to get that clip because I want to combine that with Mel Gibson in that movie, I can't remember, the Celtic battles in England where he's saying, "Hold the line! Hold the line!" What is it?

0:12:13.6 John Dues: Braveheart, probably.

0:12:16.3 Andrew Stotz: Braveheart. Yeah.

0:12:17.9 John Dues: Braveheart. Yeah. That's because when you're having a conversation like this and you talk about this leadership concept, just about everybody's going to nod along with you. But when you are actually in the moment, very few people hold the line, very few people hold the line. But at least if you have this grounding, at least you'll be more likely to hold the line because you have some techniques and some ways to sort of paint this picture that there's a firm logic. There's never certainty, but at least there's a firm logic for why you're holding the line in a particular situation. But it's very, very hard. Very hard.

0:12:58.2 Andrew Stotz: One question is, could there be such a thing like a mantra that the management team could have? Something like, "One point is not the full story," or something that they talk about in non-emotional times so that they've got it set. So when all of these numbers change and everyone looks at you, it's like, "Guys, remember, one point is not the story."

0:13:28.1 John Dues: Yeah, no, that's a really good idea. That would be a good sort of internal value or something marketing-wise that you could sort of, something sticky that would remind people of this, especially in those moments of anxiety or even panic, depending on the particular situation and the type of data that you're talking about. That's a good idea. I think the key thing is that activity is not the same as improvement. It feels good. It feels good to change something, introduce something new, new rules, new expectations, even though the system itself hasn't changed. And like you said, that's tampering. You make adjustments to a stable system based on something that's just routine ups and downs and it degrades performance. I think a lot of people are familiar with Deming's Red Bead Experiment. Less of them are familiar with the Funnel Experiment. He basically talks about when you are trying to hit a target through a funnel and you move it each time to sort of adjust for the variation from the mark. You actually, he called it going off into the Milky Way in terms of where you end up when you make these adjustments every single time.

0:14:46.1 Andrew Stotz: I thought that demonstration was so... I don't remember that he did it in the seminars that I attended. I remember the Red Bead Experiment. But that tampering is so powerful to understand the mess you can end up in.

0:15:05.7 John Dues: Yeah. And that was in The New Economics. I don't think he ever did it in a four-day seminar that I remember. But the interesting thing is generally the best choice is just to keep the funnel in the same place and keep going. But again, that's very hard. Especially let's say you're doing this as a group activity and group two, three, four, and five, you're looking over and they're making these adjustments every time, and you're just sitting there. And you're like, "Maybe they're onto something," or "Maybe I do need to move." But at the end of it, they're much farther away than you are.

0:15:43.4 Andrew Stotz: And I feel like the title you talked about, "When the Numbers Change and Everyone Looks at You," is evoking that emotion of, "Am I doing something wrong? Other people would do it a different way. Oh, they're making progress. I'm just sitting here." Those kind of emotions are the types of things that cause that tampering.

0:16:02.7 John Dues: Yeah. And then that shows up as initiative overload. You get these contradictory messages, constant course correction like in the Funnel Experiment. And the irony is you typically have a leader who cares deeply and they don't realize they're creating the very instability that makes improvement impossible. It's a tough realization. So what I would say is that when the data does change, the most important leadership move is not action, but it's interpretation. So instead of asking, "What should we do?" maybe a good first question is, "Is this shift within the range of what we should expect?" So just that question kind of slows the moment down. It shifts attention from reaction to understanding and it invites the group to look at data over time rather than point to point. It opens up this possibility that nothing is wrong even if the results aren't yet acceptable.

0:17:15.4 Andrew Stotz: Love that. Love that.

0:17:16.8 John Dues: Yeah, I think it's a really important...

0:17:17.5 Andrew Stotz: Is this shift within the range of what we would expect?

0:17:20.6 John Dues: Yeah.

0:17:21.1 Andrew Stotz: Answer's going to be "Yes, this is in the range." So next topic in a meeting.

0:17:28.5 John Dues: Right. And we've talked about this before. And it's possible when you've asked that question that the system itself looks stable, but it also may be producing outcomes we don't like. And so the key is even in those cases, reacting to an individual data point is not going to help. In that case, if you have stability but outcomes you don't like, you need thoughtful system redesign. But these sort of urgency-driven immediate fixes, overreaction, that's not going to help. That's not going to help.

0:18:06.9 John Dues: So the big thing is pausing before reacting. But that's often misunderstood. We talked about is he or she ignoring the data? Are they lowering expectations? Is that leader just indecisive? I don't think so. I think that's really what discipline is. And pausing, being that person that says, "Let's take a breath and pause here," it creates the space to study patterns rather than focusing on those individual data points. It allows leaders to separate stability from acceptability. It prevents unnecessary pressure then cascading through the system, which is what often happens. And so what I think is when you actually pause, what you're doing is protecting the people in your organization. When you do that, I think in an education system it protects teachers from being judged on noise they can't control. I think it protects leaders from... They are often then turning around and making promises that the system can't actually keep. It's sort of like a short-term thing, but you're hurting the long term. And then it protects students because they don't then undergo all these constant changes that disrupt their learning.

0:19:43.1 John Dues: So I think what a leader, a strong leader does that's different is they ask questions. What does this look like over time? Is this a meaningful signal from what we've seen before? What should we expect if nothing changes? Just some basic questions. I think resisting the urge to explain every up and down movement. And it's really at the end of the day what it comes down to is you're not trying to assign meaning to every data point, but what you're trying to do is understand the underlying system behavior. Now sometimes action is warranted, and in those cases, you're going to act in a deliberate way. When it's not, they're going to communicate that and communicate why we're going to wait in this particular scenario and why that's the responsible choice. So there's got to be this underlying logic whichever direction you're going to go. And I think if you've ever been around a leader like this, it feels calm. It just feels calm. It feels steady. And over time, the key thing is it creates this system that's trusting and then as a result, it's far more capable of improvement. It's far more likely that improvement's going to happen.

0:21:13.4 Andrew Stotz: That's amazing. And I was just taking lots of notes, but I wrote down pause, have discipline, protect employees, protect students. But I wrote down protect the aim.

0:21:27.0 John Dues: Yeah, protect the aim. That's good.

0:21:28.7 Andrew Stotz: Protect the aim of the system. Why are we here? And if we can't do that pause and look at it carefully, there's just no way we're going to achieve that.

0:21:43.3 John Dues: Yeah, no, I agree. And I think the thing is with these situations is that the most damaging decisions in schools are often made after the numbers change, but not because of the numbers themselves. Like even if they've declined, typically it's not to the point that it's catastrophic, but what's catastrophic is the series of decisions that are made as a result of the decline. And so in those situations again, this reaction feels responsible. But really what happens when you react without understanding is it creates more noise, more stress, more instability, and you still don't have the improvement at the end of all that consternation.

0:22:30.1 Andrew Stotz: Yeah, I would sum up my sum of this is the bad manager says, "That's a terrible result. Let's make it worse."

0:22:42.0 John Dues: And that's really what's happening. They're obviously not saying it, but that's exactly what's happening. Exactly. That's a really good summary. And I would kind of sum it up with three big ideas that would be helpful for listeners. I think the first one is that not all variation is meaningful. Most fluctuations actually are just routine, should not trigger action. The second one we've talked about, that overreaction creates instability. Acting on noise makes systems worse, not better. And then the third thing I would say is that pausing is a leadership skill and understanding must come before action.

0:23:30.2 John Dues: And I say it's a skill 'cause you actually have to practice it. I think you have to prepare yourself for what you're going to do when you get in front of a group and you're going to talk about results and those results maybe aren't exactly where you want them to be. You have to practice that, rehearse it. What are you going to say? How are you going to back that up? What's the logic? But I think when leaders learn to have that pause before reacting, they actually protect learning, they protect trust, and then they actually create the conditions for improvement. And I think that's the work that matters most when everyone's looking at you to make a key decision. Not easy, but certainly important work.

0:24:08.6 Andrew Stotz: That's a great wrap. I'm not going to add anything to it. John, on behalf of everyone at the Deming Institute, I want to thank you again for this discussion. And for listeners, remember to go to deming.org to continue your journey. You can find John's book, Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools on amazon.com. This is your host, Andrew Stotz, and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."