Ep. 38: Running Better Experiments at Work
About this episode:
Original Air Date: December 1, 2025Everyone says they want to “experiment” at work—especially now that AI is reshaping how teams operate—but most organizations still treat change like a project plan: analyze, design, roll out, hope for the best. The result? Fake experiments that are over-controlled and over-planned, or chaotic side projects that burn people out and quietly die. In systems this complex, you can’t think your way to the right answer, but you can test and learn your way there.
In this episode of At Work with The Ready, Rodney and Sam dig into what real experimentation looks like inside organizations. They unpack why complexity demands an iterative approach, why so many “tests” are doomed from the start, and what it takes to scaffold experiments with the right authority, resourcing, and constraints.
Downloadable tools from this episode:
The Ready’s Experiment Proposal Template
Mentioned in this episode:
Adam Grant’s astrology post
“previous experimentation episode”: BNW Ep. 62
operating rhythm: BNW Ep. 118
Even/overs: BNW Ep. 44
WIP (Work In Progress)
In this episode:
Rodney Evans
Sam Spurlin
Listen now:
Episode transcript:
Sam Spurlin:
Undermining scientific method? How dare you, sir?
Rodney Evans:
Well, this is the thing.
Sam Spurlin:
Science is the foundation of our modern world.
Rodney Evans:
Honestly, honestly. It's like, this is how we discovered literally everything that was interesting that ever happened.
[Intro Music]
Hey, everybody. Welcome back to At Work with The Ready. I'm Rodney Evans, and the man on the mic is Sam Spurlin.
Sam Spurlin:
Hello.
Rodney Evans:
AI continues to rewrite the rules of work in this moment. We feel like the future of work is here though, and now the question isn't whether or not you have a choice about adaptation, but how you're going to design work from this moment forward for what's coming.
Sam Spurlin:
Said another way, work design is no longer optional, and the teams that treat it like a side project are actively being left behind, and the ones that treat it as essential will keep up with the pace of change.
Rodney Evans:
So today we're going to dig into experimentation. I saw a post the other day from Sam Altman. He was talking about the kind of company you want to build. Honestly, not to be a fucking know-it-all, but it's the same shit we've been saying for 15 years, and having clients be like, "Are you sure?" And now, now, now, now, now, we're going to really do it, right, guys? We're going to really do it because we have no choice. So today we're going to talk about experimentation, which is one of the foundational ideas that is so poorly executed in most systems. But first, we're going to check in.
Sam Spurlin:
We are going to check in, and we're going to keep it thematic. Rodney, what is a personal experiment you've done recently or are thinking about doing?
Rodney Evans:
The one that springs immediately to mind? Everybody out there who liked Adam Grant's post debunking astrology can just close your ears, just earmuffs for the next little minute. Also, at me, because I definitely want to argue with people about this. Not because I think astrology is science, but because I think it can be useful, and I think it's annoying when people are just butt heads about it. But I digress. I have been in a month-long conversation with ChatGPT that has the backdrop of my astrological chart, but actually I just feed it content, more like a journal day to day, and talk about the weather of my life in an ongoing way. And I have found it so useful I can hardly overestimate the ways in which it has helped me process things, understand things, not react to things that I normally would. It's just been a great experiment for a few months, and I'm finding it much stickier than journaling and much more useful than a lot of other things I've tried.
Sam Spurlin:
Cool. Love it.
Rodney Evans:
What about you?
Sam Spurlin:
I bought an analog alarm clock.
Rodney Evans:
Why?
Sam Spurlin:
Because previously, I used my HomePod, which is a fine alarm clock except it has this very deep flaw, which is, you can use your voice to turn it off.
Rodney Evans:
Oh, no.
Sam Spurlin:
To use your voice, you can remain horizontal in your bed. It was becoming a little bit too easy for that to happen. So now I have to walk across the room and turn off an alarm clock. My little trick on this is that my normal alarm goes off at the normal time and the physical analog alarm clock goes off one minute later. So if I don't want to hear it, because it's horrible, I have to get up and preemptively turn it off. And if I don't do that, then I will hear it. It'll make my wife mad. That's the kind of incentive I need to get up in the morning.
Rodney Evans:
You're like a teenager. I don't remember the last time I woke up to an alarm. I just wake up. That's so...I can't even imagine what that would be like.
Sam Spurlin:
That's cool for you?
Rodney Evans:
It would be so nice to still be asleep.
The Pattern: Desire for control and lack of structure stifles real experimentation
Rodney Evans:
Okay, so let us get into this pattern. Here's how I see it out in the world. Companies say they want experimentation. Not all of them, but some of them. A lot of leaders I think understand that experimentation is the key to innovation, but what they don't have is the foundational sort of learning mindset culturally to do it. So even in companies that say they're going to fail fast or move fast and break things, we still often see an underlying fear and performative culture that prevents true experimentation from taking place. The sort of bastardized version of experimentation is something that's either overly planned and sort of no fail, which isn't really an experiment, or something that's really chaotic where people have just gone rogue and in a fairly unstructured way are trying shit.
Sam Spurlin:
Yep, yep.
Rodney Evans:
In either case, when those experiments fail, and they generally will, leaders often take that as proof that experimentation wasn't the move to begin with, and the pattern sort of repeats itself. So we're not today going to, as we have before, try to convince anyone that experimentation is the key to change. We're just going to assume that everybody listening to that podcast is already gushy with that. Instead, we're going to talk about what is required for experimentation to function.
Sam Spurlin:
Yeah.
Rodney Evans:
Yeah? Okay, great.
Sam Spurlin:
Okay. Let's get into it. I was going to start with a brief like, "Why is experimentation so central to this?" But you've made the case that everybody knows that.
Rodney Evans:
No, it's okay. You can. You still can. Maybe there's a new... I mean, we do have a lot of new listeners. Maybe there's somebody who's like, "Experimentation. What?"
Sam Spurlin:
Yeah, and we can do it really, really tight, I think. I feel like sometimes when we talk about experimentation, people are assuming that somehow we're just like these fancy weirdos who want to be different, like we're not going to use the established change management theory of the last 100 years. We just need to be special and different. I mean, maybe, but no. It's really about-
Rodney Evans:
I'm like, "I mean, yes, and..." Uh-huh.
Sam Spurlin:
It's about this idea of complexity and whether or not you believe an organization is a complex adaptive system. If you do, then the experimentation follows that, because a complex system is inherently unknowable as a complicated system, meaning that you can't just plan the perfect approach and then execute it and know what is going to happen. So if you agree that you're dealing with a complex system, the only way to have productive interactions with it is through an iterative experimental approach. So we're coming at this from a very pragmatic point of view, not as a bunch of fancy duchesses and dukes.
Parallels to R&D for pharmaceuticals
Rodney Evans:
Yeah. I mean I also want to be that. But it's interesting that you say this, and I'm glad you started here even though I said we weren't going to, because I worked for a while with a large pharmaceutical company in my time at The Ready, and I worked with a therapeutic area, and they were doing drug development. It's the closest that I've ever been to that kind of R&D. It's so interesting that in that kind of field, it is so obvious that when you're trying to create a new drug, the human body is a complex system, and you're going to have to do a lot of lot of stuff to figure out how the drug is going to impact the human body. It's so obviously in the posture, in the mindset, in the DNA of people who are doing something like drug development to have that understanding, to have the understanding that experimentation with ever-increasing fidelity and ever-increasing risk is how you get a new drug to market.
And yet, in a company, which presumably is exponentially more complex than a single human body, we're like, "What if our thing, we could just perfectly plan what all of the consequences and second and third order effects would be, and then preemptively mitigate those, and then just roll it out?" I'm like, how does anybody not see the parallel between these things? And go like, you would never do that with a drug. You would never make a two-year roadmap of how a drug is going to get into the population and then just go step by step without doing an experiment.
Sam Spurlin:
Totally.
Rodney Evans:
I mean, you could, you'd kill a lot of people, and you'd get real sued bad. So it's just so interesting to me how even though science has this very well-hewn position on how you create something that is safe and useful, so many companies just refuse to engage in that way.
Sam Spurlin:
Yeah, it is really interesting to think about it. I think the inscrutability of the human body and similar sort of biological systems I think allows us to have that mentality more easily. Whereas in an organization, because they are collections of humans and we have created a lot of human-sized ways to interact with it, we think that we can control it a lot more than we actually can. Not to make this an AI episode, but I think this will actually be an interesting thing to follow with AI as it is experimented with more and more in organizations. Where does it break up our human-sized conception of organizations in a way that actually makes organizations more inscrutable, and therefore more amenable to a true kind of iterative experimental approach?
Rodney Evans:
Right. I think that's so true.
Sam Spurlin:
All right. We did complexity 101. Yes.
Rodney Evans:
Yeah, complex. It's not a watch, it's the weather. Carry on.
Sam Spurlin:
Yeah. No, we're done. We're good.
Rodney Evans:
Okay, we're done.
What’s most company experiments are missing
Sam Spurlin:
Okay. Yeah, we're done with that part. Now we're going to talk about why and how it seems to be so difficult within organizations. It seems like, in the pattern, you kind of talked about both ends of a continuum that aren't great.
Rodney Evans:
Yeah. Yeah. So I'm going to talk about actually what I find to be missing from both, from both the overly bureaucratic and the overly chaotic side, which is, the third way here is just enough scaffolding for experimentation. So on the bureaucratic side, what you have is people who, before the experiment ever even kicks off, they want to know what the key results are going to be, how they're going to track it, when it's going to succeed, what is going to happen, and on what timeline. And the most important thing is steps in a plan, ownership, and tracking. That's not what I'm talking about when I say scaffolding.
Usually, what really good experimentation is missing, if I had to say two things, it's authority and resourcing. That's usually where I see experimentation fall down. So somebody has an idea for something that they could test, it becomes something that is either side of desk or unfunded or unclarified. Or what happens I think with a lot of good ideas is, it's unclear whether the individual with the hypothesis has the authority to actually run the experiment.
Sam Spurlin:
Sure.
Rodney Evans:
So they're like, "I think that we should change our budgeting process," and it's like, the system fights back. The system is always going to fight back. So unless you are the finance person in charge of the budgeting process, you're immediately going to run into a wall with that one. So I'm saying this because I think that when we want to run a really great experiment, we have to have a really clear space and a really clear lab to do it.
Example of The Ready’s experimentation
Rodney Evans:
I'm going to give an example from The Ready because you can pile on to this one. It's a place that I think we have done really disciplined experimentation, and it's with performance marketing. So we have a member who has a lot of experience in marketing. For a variety of reasons, we realized that it's you guys, you listeners, you're it. We realized that basically we are talking to ourselves, we're in a bit of an echo chamber, and people who listen to us and consume our content have been around for 10 years. Also, we're just talking to you guys. We want to talk to more people, not because we don't love you, but because there are a lot more people out there in the world.
So we were like, how do we essentially broaden our market? The experimentation around this has been hyper, hyper disciplined. So it started with a clear proposal, a clear timeline, a clear investment, clear role charters from the people involved, clear like, "In three months, we would expect to see this, and we're going to start tracking the data at this time to see the delta between what we believe and what's actually happening." And real clarity with me of like, "Rodney as the steward of this company, we need you to do these things, and then also to willfully ignore these other things because you're just going to put sand in the gears and be annoying." And it has worked because we gave it enough oxygen and enough time and enough space and enough clarity to know what we were testing, to run real tests, and to see real results, and to use those to steer.
Now, I'm not going to talk about the outcome because, first of all, that's still TBD. And second of all, it doesn't really matter. The idea here though is, a fair amount of work went into clarifying the Petri dish and one person, with the support of a team, saying, "This is what would be needed to run these tests well enough to learn." And then a bunch of people frankly shutting up and letting that happen. That almost never happens in companies that I see. It doesn't happen for six months, it happens for six days. And then somebody's like, "So where are we with that? Should we get a status meeting on the calendar?"
Sam Spurlin:
Yeah. Yeah. I think the way you described that just kind of clarified for me is, one could listen to what you just said to me like, "Isn't that just a ton of upfront work? Isn't that what we're trying to push up against?" It was a ton of upfront work to create the space in which a lot of very organic and trusting and quick and autonomous work can happen. I think a lot of times that is flipped in the organization. We don't really ever define where we're going to do this or what the constraints are. Instead, we are all in the mix arguing over all the ad copy.
Rodney Evans:
That's right.
Sam Spurlin:
It's like, well, that's not actually a good use of our time. We're not really experimenting with anything there other than how much we can test each other's patience.
Rodney Evans:
I think that's exactly right. To that point, I would say it is more work up front than how this kind of thing, which is really any new thing, normally gets launched to create that space. You're right. And once it's running, someone who was external to The Ready looked at that work probably three months in and was like, "You guys have done 16 months of work in three months." It's pretty astonishing in terms of the data gathered, the number of tests run, et cetera, et cetera. And this is why. When people say they want speed, it's because we weren't futzing with pixels and ad copy and arguing about palettes. We let the experts do their thing and let it run. This is where the stuff stumbles in organizations. So we hear we want speed, we hear we want experimentation, but then also, we want a lot of control and a lot of information and a lot of say in a lot of things that don't matter, and you can't actually have all of that at once.
Sam Spurlin:
Totally. This is why I find myself frequently talking about experimentation as a posture within an organization, which when I say that word, it's basically pointing to the idea that it's not just the moves you described there, but it's how all of us interact with that around it. So the idea that we're not expecting the folks who are doing this experiment to show up a week later with everything figured out, the understanding that we don't have to understand everything that's happening within this experiment, and that we trust that there are checkpoints built in. When we need to weigh in on something, we will weigh in on it. This won't be a runaway process that just runs forever. I think you need everybody around experiments to also have that posture. Otherwise, you've got this mismatch, where the people who are actively running the experiments are like, "Yeah, you want me to be doing this experiment, but really you just want me to tell you the answer."
Rodney Evans:
That's right.
Sam Spurlin:
And that is a recipe for not really learning anything.
Rodney Evans:
Well, and further to that, really what you want is me to tell you it worked.
Sam Spurlin:
Right. Yeah. The answer, we found it because it worked, not the answer because it didn't work, and we're pointing to something else.
If everything succeeds, they aren’t experiments
Rodney Evans:
And this is one of the fundamental problems with experimentation in a lot of companies, is that my super hot take is, most experiments should fail.
Sam Spurlin:
Hot take zone, zone, zone. Right.
Rodney Evans:
I think you want to run a lot of them and I think you want to kill them quickly. What that means is, we've been saying for years it's an Aaron Dignan-ism, it still slaps, "Radical change at a non-radical scale." But this is how you get that. It's like, do really weird shit, but do it really tight and not for a very long time. If 90% of your experiments are successful, they're not really experiments.
Sam Spurlin:
Yeah. I completely agree with you, and as I was preparing for this episode, I thought about something in a different way that I want to run by you and see if it tracks to your experience as well. Because I do think I agree with you, but the thing that I landed on when I was thinking deeper about this is, I see too many experiments that are kind of in the messy middle size-wise. I think it's much more interesting to do very small micro experiments that are light and easy to spin up and learn from and have very small blast radiuses, or doing something that requires greater coordination, greater cohesion of maybe a little bit of planning upfront, like the performance marketing one that you talked about that is involving lots of roles and lots of teams. Though I see those as kind of different ends of a continuum, and both are cool and interesting, and it's the stuff in the middle that I think sometimes ends up being the most challenging and the most difficult to actually execute and learn from.
Rodney Evans:
What's an example of something in the middle?
Sam Spurlin:
Something in the middle is like a couple of teams kind of half-assing something together about their workflow or about how they're working together, where it requires kind of more thought upfront about how does this actually work and interface with other parts of the organization, and we're kind of just trusting that we'll just try something and we'll learn from it as we go, which I think sometimes works. I think what it comes down to is, people don't have enough skin in the game. I think the middle one is when there's not enough skin in the game, and we're just half-assing a thing. Whereas this really small stuff, you don't need to have a ton of skin in the game. You can buy an alarm clock and see if it helps you wake up easier. The stuff that requires much more skin in the game, we've created the structure for. It's that middle amount of commitment that is the problem.
Rodney Evans:
Yeah, that's interesting. I mean, I think that my reframe on this... Because I think a lot of times that messy middle is like we're going to make a thing that we think is missing. I think that often is the experiment that happens in the messy middle that nobody actually uses. It's like, we're going to document our onboarding workflow. I often think that the messy middle of experiments is like we see something missing and we are going to make it. I think a more interesting reframe on that is, can you go and validate what the problem is with the missing thing as the experiment? Because-
Sam Spurlin:
Is it actually missing?
Rodney Evans:
Is it actually missing? I think too many experiments at the team level result in documentation that nobody gives a shit about.
Sam Spurlin:
Right. If we gave a shit about it, in most cases it would already exist.
Rodney Evans:
Yeah. If it was a real problem, it would've been documented. Also, usually, I mean documentation doesn't solve that many problems in a complex system, if we're being honest.
Sam Spurlin:
We like to think it does. It's simple.
Rodney Evans:
Yeah. We like to think it does, but yeah, what does it really do? So yeah, I think that's really interesting. I do think the altitude of experimentation is a tricky one. It's a tricky nut to crack. There are also some interesting things I've heard recently. I just saw someone, I was at the Charter conference last week, shout out to my buddies at CharterWorks, and someone on stage was saying that in her company, which I believe was an AI-native company, but in her company, there's basically a marketplace of agents. Any employee can create an agent, and they effectively get deployed and upvoted very organically within the system.
Now, that to me is a very contained way of doing experimentation, because basically I can build an agent to address my problem that I think is worth addressing. And then if you also think it's useful, you can use it. And if you don't, it doesn't matter because I still have the thing that's doing what I want. And it doesn't really require team coordination. There's a social component of it so that you're seeing which things are solving organizational problems, blah, blah, blah. That's a great use case for experimentation. What I loved about it is actually the open source nature of like, "And then anybody can redeploy these, and we can see what really slaps."
Sam Spurlin:
Absolutely.
Rodney Evans:
That feels like a really clear small things that add up to big things, but the gnarlier things like, we want to change the op rhythm with our board. It's like you probably want to do that really well one time and see what happens rather than trying to architect something for a year.
Learning and scaling successful experiments is really hard
Sam Spurlin:
Totally. And that gets me thinking about the opportunity of scaled experimentation. By that, I don't mean huge experiments, I mean lots of experiments in huge organizations. I think you just described an example of how experimentation needs to be paired with transparency. Marketplaces are tools, are mechanisms for transparency. I think the huge opportunity in a lot of organizations that I haven't necessarily seen done super well lots of times is, how does the learning from local experiments scale up into other parts of the organization?
Rodney Evans:
Dude, this is so hard.
Sam Spurlin:
How frequently are we learning from what others have done in their experiments for our own inspiration or for our own tweaks? That is I think a huge opportunity that is really hard to do.
Rodney Evans:
It's so hard to do. There's no one and done answer here. Certainly, I think having trained GPTs is a really helpful way of doing this because people can pull the information that they actually want rather than trying to consume an entire Slack feed of demos or wins or whatever. But all of that being said, it's still really hard. I think that the knowledge, I hate to say knowledge management, but the knowledge availability part of experimentation in a complex system is something that I see so few organizations really able to crack. I think it's incredibly difficult because it's not just, is the information available and pullable? It's like, do people know that? Do they know what's there? Do they know when to ask? Do they know how to apply it? Do they have time in their day? It's just like, it's a whole ball of string that, again, really relates to having learning being part of your culture, which is pretty rare.
Sam Spurlin:
Totally. Well, plus the added complexity of just because one team or one part of the organization experimented with this thing and it didn't work, the takeaway, the overly simplistic takeaway is like, "Okay, won't try that anywhere else."
Rodney Evans:
Don't have to do that anymore!
Sam Spurlin:
But that is absolutely not the takeaway. If it's a different department or a different function that has a completely different purpose than yours, maybe yeah, it wasn't a good fit for their context, but it might be for yours. Yeah, it's such an interesting thing. I wonder how and if it can really be cracked in large organizations, because I think this is one of the benefits of being a huge organization if you were able to do this really well, because you are just inherently able to test more things, more rapidly. And if that is connected to learning more and learning quickly, then you are at such an advantage. But I think it's, in most legacy organizations, which are almost by definition older, they were not built on this DNA and have not built it for themselves yet.
Rodney Evans:
Yeah, I think that's right. Like a lot of other things that we've talked about and sort of myth busted on this show, I feel like this is also one of those things that's really easy to sort of mock? I definitely have had the interaction with a trad leader who's very MBA-ish, who I think finds experimentation unserious. To your point about being fancy weirdos, I've definitely had the interaction where I think someone was, in an attempt to defend their position, trying to undermine this idea by discrediting my understanding of how work is and how real business goes. I'm like, first of all, who fucking cares? Second of all, it's just lazy.
Sam Spurlin:
Undermining scientific method? How dare you, sir?
Rodney Evans:
Well, this is the thing.
Sam Spurlin:
Science is the foundation of our modern world.
Rodney Evans:
Honestly, honestly. It's like this is how we discovered literally everything that was interesting that ever happened. But to your point about what organizations are built on, management science, which is what the Harvard Business School was founded on, the first principle is, there is a right answer. The first principles are, through analysis and research, you can be correct. So we really are, when we have these conversations with leaders who have been socialized and educated in traditional and especially in elite institutions, we are fighting against 130 years of history at this point, and trying to unlearn the idea that a right answer is uncoverable with enough analysis and debate, and into the mindset that we're not sure what the right answer is, and all we can do is tests.
Sam Spurlin:
Totally. I think maybe a way that I have sometimes taken a couple of steps in their direction and maybe break down some walls is that sometimes I think that there is a misunderstanding that what we are saying is just try random shit.
Rodney Evans:
Right.
Sam Spurlin:
We're not talking about random mutation here. We are talking about practices and ways of working and processes that we have seen work in other contexts. There's good foundational... People have thought about them, they have used them in other contexts, and we're saying, "Try this thing that has worked in other places in your context."
Rodney Evans:
That's right.
Sam Spurlin:
Not, "Try this super random thing that makes no goddamn sense, and maybe it'll work here." We're experimenting with whether an idea works in this context, not the idea itself in most cases. Not saying that you can't come up with something totally random and new, but that's generally not what the vast majority of experiments are.
Rodney Evans:
Yeah, I think that's right. What have we not hit?
Ripple effects of experiments are just as important
Sam Spurlin:
I think one thing that I wanted to say, and Jack may cut this if it doesn't really go along with the flow of the conversation, but I think it is useful to also think about the system's reaction to an experiment as data about the experiment. That is useful data.
Rodney Evans:
Say more.
Sam Spurlin:
Let's take something really simple. We change the operating rhythm, and people are asked to show up in a different way, the people who are not actually involved in the meetings. And there is some aspect of them being like... I could picture both ends of this, really positive or really negative. Like, "Who are you to tell us to meet in a different way or that our meetings are bad?" Or the positive version of that is like, "What's happening over there? I am noticing that people who are involved in this thing are happier and having a better time."
I think sometimes we limit our view of what an experiment has accomplished to the people who are in the experiment itself. So we're just asking the people in the op rhythm like, "Hey, do you like your op rhythm better?" Instead of seeing what else it changes around in the system. I mean, I think you could even think about it in more of an org design-y way, like, "All right, we changed the op rhythm with an experiment over here," and suddenly there's these conversations about strategy that have never really happened before over in this other place. What is that telling us? That's interesting. And maybe we follow our nose that way so that our next experiment has nothing to do with op rhythm. It pointed us toward this other thing that we wanted to dig into as well. Does that make sense?
Rodney Evans:
Yeah, that makes a lot of sense.
Sam Spurlin:
Okay.
Unstructured experimentation is deeply costly
Rodney Evans:
I mean, I think the other thing that is worth saying about the systemic tendency of experimentation is, they tend to accumulate inertia. My experience as a steward of a company with experimentation, and we've run a lot of them in the last couple of years in a much more disciplined way than we used to. We used to be really YOLO about experiments, and we often didn't get much from them because they were poorly scaffolded and under invested, inside of desk. It was like, "I don't know, is this person going to figure out a new line of business while they're fully deployed to a client just on Sunday mornings?"
Sam Spurlin:
Yeah. The YOLO nature of it was like, there was a very, very small chance that it was going to work amazingly well and be obvious to everybody, and we pat ourselves on the back.
Rodney Evans:
Totally.
Sam Spurlin:
And then the vast majority would continue forever or fizzle out, or nothing would really happen.
Rodney Evans:
Yeah. I mean, I can count on less than one hand the number of experiments from a bunch of years that I would say really turned into something material over time. I say this because that was also a lot easier actually. The more chaotic approach to experimentation, we're like, we're not super crisp on the hypothesis we're testing, on when we expect to see results, on what we would like those results to be in the delta, on when we're going to convey those and how we're going to measure and when we're going to steer and when we're going to shut it down and when we're going to... When we don't do that work, it's a lot easier just to be like, "There was an idea on the board that somebody opens an ice cream store. Did Joe ever get around to doing that? No? Oh, well, we didn't spend any..." It was kind of like, nothing ventured, nothing gained. But the opposite of that, which was like, "Well, we didn't get anything, but it didn't cost us anything. Who cares? Let's do it again."
Sam Spurlin:
Which actually has a huge impact, because what was invested was time and even just mental capacity, which is super rare in an organization.
Rodney Evans:
Totally.
Sam Spurlin:
Unaccounted for time and attention is maybe the rarest resource in an organization.
Rodney Evans:
Totally. But emotionally, it was easier because it was like low risk.
Sam Spurlin:
Yes. No skin in the game. Yeah.
Rodney Evans:
Again, back to the no skin in the game. So now, over the last couple of years of taking bigger swings and doing bigger things, I will say, as someone who has seeded a lot of those things, or at the very least, helped to protect the space for them, it's not easy because, first of all, sunk-cost fallacy is real around experimentation. My experience so far is, there are very few things that are clearly ready to be killed. Especially for an optimist, there's almost always some kernel of something interesting learned or uncovered in an experiment that you're like, "Should we just rescope this and keep going in this direction that's 45 degrees off?" It's very uncommon in my experience for it to be like, zero out of 10 points awarded, shut it down right now. That never happens. So prepare for the messiness that it's not going to feel black or white, and you're going to have to make a call, one.
Sam Spurlin:
Totally.
Rodney Evans:
Two, the opposite of that is also true, where when you feel like you're seeing signals and you want to double down, that also takes courage. Because also, in early experimentation, it's really rare to see undeniable proof that it's working.
Sam Spurlin:
Right. At that point, it almost stops becoming an experiment. It's just like, "Oh, okay, we do this now."
Rodney Evans:
Right. It's like, "Oh, we hit oil, so now we're just on the geyser." It does not... I mean, I'm sure it goes like that one in a hundred, but my experience is that it's rare that it goes that way. So I'm saying this because when you're a person, like I am, who holds some authority and some discernment over these things, I just want to say out loud that the messiness of it is real. The sunk cost bias is real. The searching for signal is real. It's really hard actually to do this well, and that doesn't mean that it's not the right thing to do. But counter to what I said before, you're not in a lab doing very specific trials with a team of scientists who are getting very concrete data when you're experimenting on a complex system. It does take a level of judgment and taste and opinion when you're deciding the fate of an experiment that feels pretty uncomfortable. It feels uncomfortable for me, and I think I have more tolerance for this stuff than most people.
Navigating the discomfort during experiments
Sam Spurlin:
To the extent that you think you have gotten better at this over time, what has allowed you to get better at it?
Rodney Evans:
I think a couple of things. One is not doing too many things at once. So when you're using all the dry powder across a lot of lot of bets in hopes that one of them pops, first of all, then you're trying to do the kind of discernment game in too many places. Not just as an individual, but probably as some sort of stewarding team. I'd rather actually limit experimentation in progress, have sort of a fixed number of them running at any one time, and finish one before we start the next one.
Sam Spurlin:
Right. You can use that as a forcing mechanism to decide like, "We could keep this one going, but I see my backlog. Do I want to keep this one going even/over trying this other thing?"
Rodney Evans:
That's right.
Sam Spurlin:
It forces you to get a little bit clearer other than just like, "Yeah, why not keep it going? Maybe it'll work."
Rodney Evans:
That's right. It's so easy to just be like, "Well, what if we do the other one alongside?" But there's only so much context switching and money and energy in any system. So the limitation of the number that we're going to do and also being like, "If we want to do that when one of these has to stop" is actually really helpful.
I think the other thing is, we have an H2 even/over for The Ready, which is, now, I can't remember what the exact words are, but it's basically patience with experiments in flight even/over new experiments. The fact that as a team we agree, we chose that even/over, knowing what experiments were in flight, has also helped me in the moments where I get a little fiddly to be like, "No, no. No, no. No fiddling." We are going to be patient with what is in flight until we reach a determined conclusion rather than launching a bunch of new shit because we're feeling a way.
Sam Spurlin:
Totally.
Rodney Evans:
I think those are the things: doing it as a team sport, limiting WIP.
Sam Spurlin:
Yeah, I love that. I love that. I was just going to say, to point to our even/over episode, what makes that a good even/over is I could picture a scenario where we would want the opposite, where we actually want to spin up a bunch of stuff and try a bunch of new things and not double down on things we've already been doing.
Rodney Evans:
Yeah. Totally.
Sam Spurlin:
Nice.
Rodney Evans:
I think that's exactly right.
Okay, Sam, I feel like we've covered the territory of experimentation, mostly what doesn't work. Let's give people some ideas. Why don't you start us off?
Tip 1: Create intentional space for learning
Sam Spurlin:
I will happily do that. So my first thought is broadly around, how do you cultivate more space in an organization? I think that is often what gets in the way of orgs feeling like they can do some more experimentation on the internal ways of working, because we've used this idea of the learning mindset over and over, and it's really, really, really hard to be in a learning mindset if your environment is expecting you to be in a performance mindset or in a performance context. So to the extent that space can be made, things can be stopped to create a little bit more space to talk about and run and design experiments, I'm a huge advocate of that, which is often why the first batch of experiments a team or an organization or a department might do is around, how do we create more space? What do we stop? What do we do differently? How do we reserve a chunk of every two weeks for us to deliberately shift into more of a learning mindset?
Rodney Evans:
Yeah, I love that.
Sam Spurlin:
What do you got?
Tip 2: The Ready’s Experiment Template
Rodney Evans:
This is way more tactical, but one of the habits that has really taken hold over the decade at The Ready is using an experiment template.
Sam Spurlin:
Sure.
Rodney Evans:
We've used this with a bajillion clients over the years. It kind of works every time. It's kind of awesome in every single context. And the reason-
Sam Spurlin:
It's always better than not using it. It doesn't always solve every problem, but it's always better than not using it.
Rodney Evans:
Dude, a hundred percent. I'll regularly get something shipped to me that has a bunch of other stuff up top, and I'm like, "Blah, blah, blah, where's the template?" Because the template is where I'm going to determine whether the thinking around this passes muster.
So here's what's on the template. Define the opportunity or the tension. So what are we solving for? What do you need? Be specific. You only want one tension per experiment proposal. So don't be like, "Here's 50 things that are wrong." What's the actual thing that you're solving for? Relevant background, give a little context about the situation. Proposal. What are you actually proposing to address the tension? We love this. Then other options considered but not recommended.
Here's what is so rad about this field, is that everyone who has ever sat in a meeting of any kind who's heard a proposal of any kind, the immediate response is, "Did you think about X?" Here's your opportunity to be like, "Yeah, nerds. I did think about X, and I wrote it down here so you don't have to ask." Now, on the off chance that you have an idea in that meeting that is not captured here, fantastic, but my experience is, nine times out of ten, they in fact did think about X before I had a chance to be annoying about it.
That's the content of the experiment that's being proposed. Below that is additional context that I find incredibly valuable. So there's facts. What do we know? Data points, constraints, et cetera. There's assumptions. What assumptions are we making? There's constraints. What constraints must be considered? What's non-negotiable? That could be like, "We only have $10,000 for this," or, "This has to be done by the end of the year." Risks. What are the risks if a key assumption is wrong? So calling back to the other field. Dependencies, impact to the system, other roles or people, what are some possible consequences of the proposal. And decision makers. Which person, role, or group has the authority to make this decision?
Okay, those are the fields of it, and there's prompting questions for each one that I didn't talk about. But what I find, this gets you without actually... It takes way, way less time than starting from a blank sheet of paper to try to explain something, is it really makes explicit what we know and what we don't know. That is incredibly useful because we're basically trying to get a group of people to say this is safe to try even though we don't know a bunch of stuff.
Sam Spurlin:
Right. Yeah.
Rodney Evans:
And that muscle in and of itself is so weak in most organizations. Usually, the conversation is, "How could we know more?", rather than "Knowing that there are these gaps and assumptions and risks and whatever, can we do it anyway?" So that we don't spend more time trying to know more, we can immediately start trying to learn more.
Sam Spurlin:
Yeah. Yeah. I've experienced the difference between sitting in a decision meeting where there has been a proposal filled out ahead of time that we've all read and where there hasn't been. And boy, howdy, is it such a better conversation when that proposal template has been thought through and filled out, and people have been able to read it ahead of time and may be even already dropped some comments in. The quality of the conversation is so good.
Rodney Evans:
Yeah, it makes my teeth itch when a client just sends me a document that's like 12 pages, that's like, "We want to talk about this at the offsite," and I'm like, "Oh my god, how? How are we going to do that?" So yeah, I love a template.
My one other pro-tip about using it is, decision rights and authority is just sort of borked in most places, so our little template assumes that you know who needs to make the decision. I get that that's easier said than done in most places. My sort of hack when I'm creating a proposal is, if authority isn't clear, I try to get the people involved too. One, we'll actually have to work on the experiment to make it happen, or a representative of that. Two, the person or people who will have to fund it have an eye on the monies or the resource allocation or whatever. And three, someone who's going to be involved in the longer term maintenance.
So if I'm like, "We should make a course about AI," it's like you should probably have the person who's going to have to make the landing page and create the workflow and think about the whole thing, not just the inception of the idea, in order to get consent, even if they're not all involved at the very beginning. Because what you don't want to do is run an experiment that turns out to be really, really rad, and then have whoever's maintaining that work, whether it's ops or technology or HR or whoever, be like, "Yeah, cool. We don't have any capacity to do this. So why did you make it?"
Sam Spurlin:
Totally, or you made some decisions early on that didn't seem like a big deal to you because you aren't an expert in this domain. Actually, we've kind of borked ourselves by not making this alternative thing.
Rodney Evans:
Yeah, exactly. That's it.
Sam Spurlin:
Yeah, totally. I love it.
Rodney Evans:
What you got?
Tip 3: No experiments for other people
Sam Spurlin:
Okay, I've got one much more tactical than my first one. So this comes from the observation that every time I work with a team who is thinking about ways of working experimentation, op model experimentation, experimentation focus on the organization itself, far in away, the way things go off the rails is that everybody has experiments for other people to do. So everybody has come up with experiments left and right, and they're never about what you actually have any sort of influence or control.
Rodney Evans:
I love it. Yeah.
Sam Spurlin:
I've made the rule that in these workshops, you're not allowed to make an experiment for anyone else without being in conversation with them.
Rodney Evans:
Great. Yeah.
Sam Spurlin:
So really, what that means though is just start with where you are. If you're a senior leader, you have more kind of freedom of movement and greater influence to propose experiments because you do influence more. But even as an individual contributor on a team, there are plenty of experiments that you can steward and own, even if it's purely just the individual, how I show up to things or how I do my role. All of that is valuable. Building the muscle of identifying the levers that you can experiment with that you actually can reasonably reach, it will serve you and your team and your organization so much better than just blue sky solutioneering experiments for everybody else.
Rodney Evans:
Solutioneering is a new word that I like a lot. It feels like the combination of mountaineering and making solutions. Solutioneering, folks, you heard it here.
Sam Spurlin:
That's right.
Tip 4: Prepare yourself for disappointment
Rodney Evans:
Okay, my last idea that's not really an idea at all, but is the warning label on experimentation is, if you're doing it right and you're having the moments of reflection, and you're having the cadence of check-ins and retrospection, and you're really looking clear-eyed at the data to see if this thing has legs, you're going to have a lot of failure. Also, what is going to come with that is probably disappointment. I'm saying this as a PSA because most of us, especially in professional contexts, try to avoid disappointment at all costs. Maybe even more than failure, we try to avoid disappointment.
So prepare to kind of hate this, because no experiment gets launched because everybody thinks it's a stupid idea. Everybody launches an experiment with a twinkle in their eye, and when it doesn't work out, it's a bummer. You can't avoid that, and it's not the reason not to do it, and it's not the reason to massage it into something that is inevitably going to work, because again, that's not an experiment. But prepare yourself and your team that's launching it for the reality that you are likely going to have to kill your darlings, and you are likely at some time going to be confronted with your past self who had a lot of belief and optimism around this thing, and is now like, "I got to take it out back and shoot it." It sucks. That part is really hard. So have the kind of systems, conversations, support, whatever, around that allows that to happen. Because if you're avoiding that at all costs, you're never going to do anything interesting.
Sam Spurlin:
Totally. Maybe the glib thing that I was going to say is like, "Well, that's why the true measure of an experiment is not whether or not it worked, but whether we learned from it," which, yes, is true and also is hard-
Rodney Evans:
We love to say that.
Sam Spurlin:
We do love to say that.
Rodney Evans:
I love to say it too.
Sam Spurlin:
And it's one of those things where you can say the words and agree with the sentiment. And when the thing that you conceptualized and thought was going to be great ends up not being that, and yes, we learned from it, yeah, of course, we'll capture those lessons and we'll share them, it still sucks. I think over time, you can start to divorce your identity from like, does this experiment work or not? And really focus on, what is it teaching us and what are we going to do next? But you have to be in a particularly, I think, receptive environment for that to actually be cultivated and grow over time, which is not always the case.
Rodney Evans:
Yeah, that's right. All right, Sam.
Sam Spurlin:
All right.
Rodney Evans:
I think that we've given them what we can on experimentation. Now they just have to go do stuff.
Sam Spurlin:
We've done our job, it's out of our hands now.
Rodney Evans:
We did our job. Love it.
Sam Spurlin:
All right. We are always looking for new topics for the show, so if you have an organizational pattern that you're having trouble changing, or for this episode in particular, if you have an experiment that you ran or participated in in your organization that either went really well or really poorly, just want to tell us about it, I would love to hear those as well. You can send them to podcast[at]theready[dot]com.
Rodney Evans:
Nice. This show is engineered by Taylor Marvin and produced by our vacationing friend Jack Van Amburg, who I kidnapped last weekend in Los Angeles. More on that later. At Work with The Ready is created by The Ready where we help organizations around the world change the way they work. Thank you so much for listening.