Building an AI product taught Teresa engineering tradeoffs

Show Notes

Product Discovery coach, teacher, and author Teresa Torres joins Peter and Jesse to explore the messy reality of organizational change and cross-functional collaboration. She discusses why external coaching has limits, how individual contributors can drive change within resistant systems, and what the rise of AI means for blurring roles between product, design, and engineering teams.

Learn more about Teresa Torres and her teaching at https://producttalk.org/
Learn more about Jesse James Garrett and his coaching and consulting at https://jessejamesgarrett.com/
Learn more about Peter Merholz and his new digital book “Design Org Dimensions” at https://petermerholz.com/

Transcript

Jesse: I’m Jesse James Garrett,

Peter: and I’m Peter Merholz.

Jesse: And we’re finding our way,

Peter: navigating the opportunities

Jesse: and challenges

Peter: of design and design leadership.

Jesse: On today’s show, Teresa Torres, product discovery coach and educator, and author of the book Continuous Discovery Habits, joins us to look at the changing landscape of digital product development from the product side. We’ll talk about what a coach can and can’t do for organizations and leaders, where product and design leaders get off track in driving meaningful outcomes, and what the new wave of generative AI means for the relationship between product and design.

Peter: Hi Teresa. Thank you so much for joining us.

Teresa: Thanks for having me. I’m excited to do this.

Peter: So I’m gonna start with a very easy question. Well easy to ask. I dunno how difficult it is to answer, which is: just how do you introduce yourself these days? I’ve known you now for 15 years, and so I’m curious how you’ve evolved since we first met.

Teresa: Yeah, I mean, I still introduce myself as a product discovery coach. I’m now an author, so that’s new in the last 15 years. But you know what’s funny is I don’t do a lot of coaching anymore. I mostly do a lot of curriculum design. I still do coaching through our office hours and things like that, but it used to be, my primary mode of engaging with people was I coach teams in depth over time. And that’s kind of changed. I now do a lot of curriculum design and we, our primary delivery mechanism is courses.

Peter: Say a little bit more about that. When we worked together I was embedded within design teams and, you know, you were coaching us through these product discovery engagements at both OpenTable and then again at Snagajob, and there was a curriculum component to it at that time, I remember, but I’m curious how that’s shifted or evolved for you and, why more focus on curriculum than maybe the kind of hands-on coaching?

From Coaching to Curriculum Development

Teresa: Yeah, a few reasons. So yeah, I started out as a coach. What that meant was I worked with a cross-functional product team for roughly 12 weeks, sometimes a little longer for the teams that needed it. But our goal in those 12 weeks was just to build really strong discovery habits. It was everything from: how do we collaborate as a cross-functional team, what does a good customer interview look like, how do we run quick assumption tests that we can rely on, h ow to start with clear outcomes, kind of the all the habits you see in Continuous Discovery Habits.

And I think just ’cause of the nature of my personality, I get bored when I do the same thing over and over again. And so I started creating a formal curriculum behind my coaching practice, ’cause I just got sick of telling teams the same thing over and over again. So then they would basically go through a content module. They would try to apply it in their own practice and then they would come to coaching.

And what I found eventually was that the things that we would cover in coaching were more like, here’s what’s hard in my unique organizational context, and not, here’s what’s hard about the skill of interviewing. And that’s because the content was getting good enough that it was helping them build their skill, but it didn’t help them with the messiness of, like, but my sales team doesn’t want me to talk to a customer.

And so I realized that one of the challenges with coaching is you need a leader who’s willing to invest in it. It’s kind of a high value, which means high cost, service. And I just wanted to get some of the skill development out to more teams.

And so with online courses, you can offer it at a much lower price point. A lot more people can engage. We still do, I say we ’cause me and some of my instructors still do, coaching with some of the organizations that we go a little bit deeper with. But I would say primarily I do curriculum design to support our suite of courses.

The Product Discovery Mindset

Jesse: So tell me a little bit more about what you mean when you talk about product discovery and who you work with inside these organizations to support the development of this practice skillset. What would you call it? It’s almost a mindset, isn’t it?

Teresa: it is a mindset. There’s several mindsets. There’s several skills… I mean, there’s like the academic term of like knowledge, skills, and abilities, KSA. It sort of spans a lot of things. I think the big thing for product discovery, I try to keep this really simple. How do we know we’re building the right stuff?

And every team in order to build something, has to make decisions about what to build. How do we know those are the right decisions? How do we have good feedback loops with our customers to help steer us along the way? I know since your audience is a lot of designers, I will be upfront about a lot of these practices came from design, but came from a particular type of design.

Jesse, I know we’ve talked about like, and both of you… Design agencies and like these practices of research and human-centered design and feedback from customers.

I think what’s changing is, in the early days of the internet, we didn’t have the right people on our teams to do that research. And now companies are starting to invest in cross-functional teams that are building those skills. And so we’re bringing it in-house. Product teams are doing it themselves, instead of going out and hiring research firms or going out and hiring design firms. And so it’s led to a more team approach. It’s led to a more, just, cross-functional approach.

And then I also think the pacing of it has changed quite a bit. So when we outsource something, we tend to hire a company for a big project. When we’re doing it internally, we can do it more continuously because we’re working on a product more continuously.

Jesse: Do you engage with design teams in your practices? Is that a part of what you do or are you really focused on…

Teresa: yeah.

Jesse: …the product management side?

Teresa: So in my coaching, I always coached the cross-functional product trio. So it was typically a product manager, a designer, and an engineer. Trios look different at every company. Sometimes they’re quads, sometimes they’re quints. Whatever the cross-functional team is, what roles you have.

So, like, some teams that I worked with had a user researcher as the fourth person. Some teams had a product marketing manager as the fourth person. Some data products have a data scientist, sometimes in lieu of a designer, sometimes in addition to a designer.

So when I say product trio, it’s really just the cross-functional product team. Whatever roles are involved and however many people that is, once you’re more than five people, it gets hard to move quickly. I would say generally it’s three to five people. And then we see the same thing in our courses. So we have product managers, designers, user researchers. And even lately, more engineers enrolling in our programs.

Peter: I am curious how you think things have progressed, because in my interactions and conversations with my clients and other design leaders, it feels like so many organizations are still struggling to do just the basic blocking and tackling of things like product discovery or dual track agile or contemporary product development, or whatever you want to call it.

There’s still a lot of PMs or POs getting requirements from the business and funneling it out to a group of people and saying, go build that. And I’m wondering if, and maybe now given your perhaps broader perspective, ’cause you’re engaging with more people through teaching than you probably could through coaching, like, are we still bending towards progress as a whole or are we, it feels at times, like, we’re kind of stuck or sometimes we go backwards. Like, what’s your take on, just, like, how things are going in terms of organizations adopting continuous discovery, product discovery, these practices.

The Unevenness of Change

Teresa: I think we’re still making progress. I think it’s uneven and there’s some two steps forward, one step back happening. So it can feel like in a single moment maybe we’re not making progress. But I just think back, like to the beginning of my career, okay, so I took my first full-time job in 1999. At that time, I graduated a very naive student of HCI.

And I really thought this was just how business worked. And as you all know, this is simply, it’s still not how business works. But I’ll tell you, my first job, you wanna know what my title was? It was application software developer. Do you know what I did in my job? I did interaction design.

Peter: Right.

Teresa: People didn’t know what interaction design was.

They didn’t hire designers, they hired people that built applications and software. And so even though I was a full-time designer, my title was application software developer. We have certainly come a long way. I’ll also tell you that, like, at that time, I think I was involved in a year in maybe like two or three usability studies because we had to rent a facility with a two-way mirror and it took all day and we spent a month recruiting the right people, right?

Like if we just remember where we came from, like we have made tremendous progress. Now, I also, like, during that same time period, the web was blowing up everybody and their mom wanted to be a web developer, which means everybody and their mom wanted to be a designer.

And now we say things like designers own research and designers own that. No, some designers have that background. Some designers do not. Some product managers have that background. Some product managers do not, right? And so I think some of this, like, it’s just as messy, it’s way messier than we want it to be.

Here’s what I do see. I see way more companies recognizing if we wanna make good decisions about what we build, the customer needs to be part of that process somewhere. I think it’s hard to find a company that doesn’t agree with that statement. Now, does that mean they’re resourcing their teams to do that? Well, not necessarily. Does that mean when the economy shrinks that their founders, some very famous vocal founders, don’t start talking about we gotta go back to command and control and this is the right way to do things and we don’t empower our teams anymore, right?

Like there’s always false starts, there’s always ups and downs. But I think the long arc, we have made tremendous progress. I’m still blown away when I meet a team that is talking to a customer every week. We have a whole industry of discovery tools. Of unmoderated testing platforms, one question surveys, all sorts of testing tools that like literally would not have been able to exist ’cause there was no market for them. Whereas now there’s a huge market for them.

Jesse: I’m interested in the cross-functional collaboration piece of what you described. You know, I often feel like everything of any strategic significance in an organization usually came out of a conversation among three or four or five people. Tops. It’s rare that it’s a mass of people coming together to actually drive something meaningful.

And I wonder about the different perspectives that you have to reconcile as part of that process, and what it takes to create a sense of being, actually just being on the same team, you know, across these different functions. I hear from so many leaders who feel that they are caught in incentive structures that incentivize different behaviors and different outcomes cross-functionally.

So design’s chasing design’s goals, and product is chasing product’s goals, and the organization doesn’t really have any interest in reconciling those. And so we have to reconcile them on the ground, face-to-face in those three to five person meetings.

And I’m wondering about how you’ve seen that done effectively.

“We suck at collaborat##ing”

Teresa: We don’t, I mean, I think if the outcomes aren’t aligned, they’re not gonna work as a team. The good news is I do see way more companies giving cross-functional teams a shared outcome way more than I’ve ever seen in the past. Like, I think we are making progress there. Now, of course, each functional role has other goals or other health metrics they care about, right?

Like, a cross-functional product squad isn’t gonna have a shared outcome of like site uptime, but your engineers better care about site uptime, right? And there’s probably similar analogies for every function.

I think here’s the reality. We suck at collaborating and we see this like up and down the organization. Like our executive teams are totally dysfunctional ’cause we don’t know how to collaborate across functional roles. Too many of us are taught from day one in business, be territorial, have clear boundaries between your role and the person next to you.

And this ladders all the way up the organization. It’s why the head of sales only cares about closing the next deal, and the head of product only cares about how we’re gonna serve the whole market. And the head of design only cares about, but it’s not delightful enough.

I know these are like platitudes and I’m not trying to dismiss any of these things, but like, it causes a lot of dysfunction and, like, if we don’t let go of those things, we will actually never collaborate.

The good news is, I think, at the individual team level, it’s easier to do this than at the executive level because at the team level, your scope is narrower. You’re just trying to build a thing. It’s three to five humans where you can build strong relationships. There’s not as much at stake, and it really is just like, how do we learn about each other and learn about each other’s strengths and figure out who should do what when.

I think it does require that everybody let go of maybe owning some of the things they think they should own all the time. And so here’s what I mean by that. Let’s say a designer wants to get their hands dirty and like play around with Cursor and implement some of their design. Should a engineer on the team that usually does front end development be threatened by that? I don’t know. Maybe not.

And like when the product manager uses Lovable to like play with the feature before it goes to design, should the designer be threatened by that? I don’t know. Now if we’re shipping it without it going to the designer, I have a problem with that. And like, ideally, all three of these people are like ideating together and playing with some of these things together.

But I think, especially with generative AI, like, the boundaries between our roles are gonna blur even more than they already have. And I personally have always been a boundary spanner. Like I’ve been a designer, I’ve been a product manager. I do some coding. Like, I really think the more we overlap in roles, the better we’ll work together.

But it’s hard because our egos get in the way, right? Like, I wanna own that decision. I wanna do that piece.

Peter: Yeah. Egos get in the way, as Jesse said, incentives get in the way, right? We are rewarded as individuals, not as teams. and I’m wondering on that front, if you’ve identified a set of qualities or conditions that indicate to you that this group of people is more likely to adopt practices in a healthy way. Like what are those things that you look for that tell you these folks are ready to embrace these new ways of working and, and succeed? And what are the, maybe, warning signs that like, yeah, I can teach you new ways of working, you’re, gonna be stuck because the problem isn’t one of process. it’s something else.

Teresa: There’s definitely some organizational stuff that I look for. So like, especially when I was coaching in my sales process, I would tell the leaders that I was contracting with like, do not bring me in to coach your teams. The rest of the organization isn’t ready for this because here’s what I saw really early on in my coaching practice, is I would teach cross functional teams how to do this work, and then the bosses would still say, no, just build what I’m telling you to build. What do you think all those cross-functional teams did? They went and got new jobs. They learned a more fun way to work, right? And the first time that happened to me, it was really painful. I was like, oh man, this company just paid me a lot of money to train their teams and they all left. And so I started being really upfront about that with leaders in the contracting process.

Like, don’t empower your teams unless you’re ready to empower your teams. So there’s a lot on the organizational side, on the team side. We have to break a lot of bad habits and we, a lot of this is just trust. The team has to trust they’re gonna be rewarded for these new behaviors.

And I think that’s the hardest part. Is that, like, if the rest of the organization isn’t on board, all it takes is one middle manager anywhere in your organizational hierarchy that affects that team that can break that trust.

So if a designer is hearing from their design director that they shouldn’t be brainstorming ideas with their cross-functional product team, they should be doing it in a design studio. Or more often what we hear, if the engineering lead is hearing from their director of engineering that they shouldn’t spend any time on discovery because they need to be increasing their velocity of code it’s gonna be a huge problem.

We see the same with the product manager. If the product manager, like is still getting pressure everywhere from the organization to just build what they’re told, then all this stuff we’re telling them feels like a charade. It feels like the flavor of the week, we’re not really gonna do this.

Peter: When I think back on our time together, and more of the discovery coaching mode, it was a lot about process. It was a lot about, kind of, steps along the way. And I’m wondering how your approach has evolved to accommodate these questions of trust and relationships and what do you teach people, or how do you help them better relate to one another and build that trust?

Think ‘habits,’ not ‘process’

Teresa: Yeah, so the first thing is I look at it more as a collection of habits than a process. That’s why habits is in the book. I think that’s an important distinction because there’s a process to how we teach it, but hopefully the teams that we work with take away, you’re not doing every step, every time.

We talk a lot about, we’re putting tools in your toolbox and you have to decide which tool is relevant in the current situation. And that I think is really important, ’cause so many people want a process to follow or a recipe to follow. And we all know life is much messier than that.

Peter: I’m wondering if, as part of your teaching now, right, ’cause you’ve talked about the importance of trust and the importance of the relationships within the team members, are you addressing that dynamic head on in your teaching? And, and what are the things that you found help people better connect with one another?

Teresa: Yeah, we do a lot of this and you know, we, there’s this pattern I really love. So first I’ll explain the problem this pattern solves.

A lot of us in business, we’re taught to advocate for our point of view. Which means we don’t really listen to other people’s points of view. We just go into a meeting trying to win the conversation. And even people that aren’t competitive, like this is what business has rewarded. I’m not suggesting this is like an individual problem. I think this is like the culture of business is that there’s a lot of advocacy going on.

So the first thing we try to do is we try to break that. We try to say, look, in all of our courses, we follow this pattern where people work in groups. So we’re trying to simulate a product trio. We give them a homework assignment that they do individually and then they come to class and their goal is to redo the homework assignment in the context of their team.

And we follow this pattern of: first, share your individual perspectives on the, on the assignment. So they’re each sharing their individual work with each other. And we teach them don’t listen for what you think they did wrong or right, just listen for what’s different. Be curious. This is about exploring the different perspectives on your team.

So try to let go of this idea of like, my version is the best version. I need to win. But listen for, across all the different perspectives,. what you like about it, what’s better, what you can take from it.

And then we have them co-create a team version. And what we find sometimes in week one, they’re not very good at it. It’s hard to let go of this, I need to win. But when they do it four or five times, they start to learn like, oh, our team version can actually synthesize all the best parts of the individual versions.

And this is very different. Even if, like, what we see a lot of teams intuitively do is, they wanna come to the meeting and start the work together, but then we get all this groupthink problems, the most dominant personality dominates the work, right?

So like, it’s really important we start with, do the work individually. Take the time to exploit all the different perspectives where you truly understand the different perspectives. Then co-create a new version. And when we see, when teams practice that, and get good at that, they realize it’s not about, did they have the best option. It’s about collectively, we’re able to take the best of each of us and co-create a great option.

Jesse: I’m curious about how that collaborative dynamic potentially plays out beyond the team level, especially looking at the leadership level. You touched on earlier how if you’ve got dysfunction in your leadership relationships and your leadership collaboration, that’s gonna cascade down into the cultures of those teams.

And the people of those teams are gonna model themselves on, you know, the priorities that are being set by, and the messages that are being sent by the leaders that they report to. And I’m curious about cross-functional leadership collaboration, and creating that alignment in ways that can have a positive cascade effect in ways that can actually help entire teams get along, and not just individuals.

Teresa: Yeah. Honestly, I have no idea. I’m gonna share, like I have no idea how to solve this problem. I’ve been on executive teams. I’ve been a CEO where I was responsible for an executive team. I have coached executives. I’ve done work like alignment workshops with executives. I don’t do that work anymore. I don’t know what the answer is.

I’ve read all the books, like The Five Dysfunctions of a Team, right. Like I, I’ve read the books, I know the theory, like I’ve worked with, I’ve seen other great executive coaches come in. I think here’s the challenge at the executive level, everybody who is an executive has by definition had a lot of success in their career and that success came from what they’ve done historically.

We all know it’s The Who Moved My Cheese book. What got you here won’t get you there. right? Like, it’s really hard for us to let go of what has worked for us in the past. But I think the challenge is, there’s always a little bit of truth to this in every role, but I think especially the executive role, to be a good executive and to contribute to a well-functioning executive team, you have to let go of your needs, if they come at the cost of the whole business’s need.

This, like, skill of thinking across the business and being able to think from the different perspectives is really critical. I don’t even know many CEOs that are good at this, and they’re the ones that like, it’s their job to be good at it. So I think what we’re bumping up into at the executive level is this is a really hard thing for humans to do. I mean, if I had a silver bullet here I would probably do more executive coaching. But I, I, it’s messy and I think it’s hard and I don’t particularly enjoy it, which is why I work at the IC team level, that is a scope that maybe people are more idealistic and more willing to change and, a little more coachable.

Peter: Following a bit of what Jesse was saying, and you saw me flipping through your book, in order to adopt the kind of continuous discovery approaches that you are teaching, people internally would need to embrace some flavor of change management, right? This is a new way of working, thus it is change.

And one of the frustrations I’ve felt, largely on part of my clients who are in these organizations that are going through a product transformation. adopting the product operating model. They’re learning to do product discovery. And, a frustration that I hear from them is, they’re shown how to do it. Like, the glorious future of, just work like this, but they’re stuck in their existing day to day.

And it’s not like they couldn’t get to that point, but it’s a process to get to that point. It’s a change management process to get from where they are today to this glorious future.

And oftentimes the leadership who’s spent on the training or whatever, is just like, you should just be working in this new way. But the teams, feet on the ground, are, they’re not. The product owners aren’t comfortable doing customer interviews, and liked being able to just tell everybody what to do. They don’t wanna work three in a box or whatever.

Like, there’s an evolution that’s required, and I’m wondering how you teach, how you think about, what you counsel your clients or students on, in terms of wherever they’re beginning, the slog, the necessary slog it is to get from that point to this glorious future and, how you help them navigate that change.

The lie at the heart of most change management

Teresa: Yeah, this is a really big question. I’ll share.

I went back in 2011 and I got a master’s in organizational change. Here was my takeaway. Organizations don’t change. People change. That’s like the headline of my takeaway, right? There’s a really great book called Managing Transitions, William Bridges, that really talks about what an individual goes through when they’re going through a big change.

And then he talks about it as like, almost like grieving, like there’s three stages you have to grieve for what you’re losing. Then there’s this messy, neutral zone, and then eventually you step into the new future. And I like Bridges’ writing on this because most change management people, the big names, John Kotter, there’s several other big folks that write in this space, they talk about change management as a process that happens to the organization, but I have literally never seen that work, right?

You can bring in the McKinsey’s and pick your other favorite consulting firm and they’ll create a Gantt chart with swim lanes and tell you here’s where you are in the change.

Like it doesn’t reflect reality. In fact, even Snagajob, Peter, is a good example of this. Snagajob had all the right things in place. They had a product leader and a design leader that supported the change. I actually don’t know who the engineering leader is, which maybe that’s one of the challenges.

They invested in coaching all of the teams went through coaching. One thing I’ve learned since we worked together at Snagajob is maybe we could have done more at the middle management layer, r ight. Like we’d hit the teams, the executives were on board. And even so, even with a lot of the right pieces in place, not all of the teams moved the same amount.

So why does that happen? Well, every individual is going through those three stages very differently. Some people don’t ever grieve, ’cause they’re not willing to let go of what they’ve been doing. They refuse to change, right? Some people step into change, they try it on, and they really struggle with that grief.

Or they get through the grief and they’re in that messy, neutral zone, but they’re struggling to see what the future looks like.

And so I think something we dramatically underestimate is how much work leaders need to do to guide every individual in the organization through the change. This is another reason why I moved from coaching to courses.

So in courses I can teach you skills. Courses Courses are very good at teaching skills. What I struggled with as, a coach when I was an outsider, it’s hard for me to guide you through the change because I’m not your boss. I’m not your leader. I’m not setting the culture. I’m not able to do rituals that reinforce it. I can’t combat a middle manager that disagrees, right?

I think I really believe in something Marty Cagan says, which is, “one of the primary responsibilities of a leader is to coach their teams.” To coach the people that report to them. I saw this as a huge limitation of being an external coach.

I think as an external coach I can help with skills, but there’s so much of this messy internal stuff that if you’re not managing the people you’re coaching, it’s hard to be successful.

I’ll give a really concrete example of this. Early in coaching, we helped the teams get access to customers. This is actually something something Snagajob was already really good at.

When we were working with the teams there, they were already using Ethnio. They’re already recruiting people directly from the product. Actually, still to this day, 12 years later, I still use Snagajob as an example. But I still had teams show up three weeks in a row saying they didn’t do an interview. They just didn’t do it. And I can spend three coaching sessions on where the resistance was coming from and why, and to help them break down barriers. But if they didn’t wanna do it, there was nothing I could do. I’m not their boss. I’m not there with them.

The importance of coaching

Peter: Right.

One of the challenges, that you’re identifying, it sounds like it happens in product, it definitely happens in design, which is design leaders don’t know how to coach. That’s not something that they’re taught, that’s not something that they realize. They know how to direct, but that bringing up is something, is a skill that’s kind of gotten lost.

You almost want, I’ve thought this about this for a while and, what you said is kind of reintroducing it, right? You almost want like a, like a master/apprentice relationship between a leader and their team in terms of really helping them understand the craft.

And I don’t mean it in the designerly way of craft, just like what are the, to use your term, habits, practices, whatever it is that you need to embrace in order for that next generation to be effective. And leaders are rarely equipped for it. And if they are, the other problem is that they’re almost never given the time to do it, right.

I think something that’s not recognized is that kind of coaching. I mean, that takes a lot of time, right? It’s not just…

It

Teresa: does take a lot of time.

Peter: …Yeah, it’s not a one-on-one every other week or so, ” How is it going?” It’s rolling up your sleeves.

But then, leaders are doing more and more IC work, but that’s not coaching, right? Like, that’s, not leveraged. It’s a kind of labor that I don’t think is appreciated because it’s not obvious that it’s driving outcomes, kind of, directly.

Teresa: Yeah, I think is a huge problem. I think most leaders in any discipline don’t know how to coach. Why would they? That hasn’t been a part of the job description. But I think if we’re going to empower teams, now coaching is required and I think, ideally, like eventually because you empowered your teams, you now have more time to coach.

The challenge is the transition I have to coach my team so they can be empowered so that I have some free time. I’ve been thinking about this a lot and I’ve been talking to a lot of product leaders about this very topic this summer. Leaders need better resources for how to get good at coaching and what does that look like and what does coaching their teams even mean?

Jesse: Well, speaking as an external coach, I feel like I have to defend the value of external coaching a little bit here. Just in that the work that I do is one-on-one work. It is also at the leadership level. So in a lot of ways what I’m bringing as a coach is not really about change management or organizational transformation so much as it is helping a professional look at their own practices and supporting them in their own evolution and their own change management, so to speak.

Teresa: Yeah, let me clarify. I actually think that there is a role for external coaches and especially at the leadership level. I think there’s a difference between, I personally have a coach who helps me see my own stuff and helps me get outta my own head and hold space for me and is a sounding board and ask me questions, I think there’s always space for that, and I think all the way up and down the organization, even at the IC level,if we could get to a point where everybody could have a coach, I think that would be an amazing world, because we need way more reflection than we naturally do.

I think what I’m getting at and like what I experienced as a discovery coach is a little bit different. It’s an organization is trying to get from point A to point B, which means all the teams in that organization are saying, you need to do B, you need to get to B.

And so you bring in an external coach and you help them with like, what does it mean to get to B, how do I get there? But if there’s stuff happening in the organization that’s competing with like, yeah, we’re trying to get to B, but we’re gonna do C along the way, we’re not really, not really on the way to B.

That’s when, like, how effective that external coach can be is really limited. ‘Cause there’s so many competing forces, and this is where I think, like, if the leader was coaching those teams on how to get to B, The leader would see oh, C is interfering, I need to go remove that obstacle, right?

And like as an external coach, I tried so hard, I would meet with leaders and be like, here’s what I’m seeing from your teams. Here’s the obstacles. But it’s a little bit like when we tell somebody about our research, they don’t care unless they’re involved. I think with change management in particular, this is where leaders have to be directly involved in coaching their teams. Coaching is a terrible word ’cause it means a lot of different things.

I think it’s different from what I would think of as traditional executive coaching.

Peter: I face the same thing. Sometimes I’ll get a director of design reaching out to me who wants my help, but if their boss is the head of design, I’m like, I need to work with them. Like, ’cause I’m gonna work with you and what I might say to you might be in conflict with what your boss is telling you, and, right? And there’s that dynamic there that I think is what you’re getting at and the risk that we’re setting up by having too much of that kind of directive coaching be external, right? Because the team needs to own it. That leadership chain needs to own it.

Teresa: I even see this at the product leadership coaching level, right? So, would I coach a product leader on how to bring the discovery habits to their organization? You know, the first question I would ask is, does your CEO want this? Right? Because if the CEO doesn’t want it, how effective are you gonna be?

Jesse: Yeah. Yeah. It applies at every level of the organization. What I’ll often say to prospective clients is, the more power you have, the more I can help you activate. And if you are in a position in the middle or the lower middle of an organization, yeah, we’re gonna be, we’re gonna be resorting to some gorilla tactics to get things done, because you don’t have necessarily the latitude or the permission to just make things so.

Two conflicting, competing thoughts

Teresa: There’s something I’ve come around to. It’s gonna sound like they’re competing thoughts, but I think there’s power in the fact that they’re competing thoughts.

So the first one took, I mean I’m probably still learning this, but it definitely took like every day of my full-time employee experience to like, actually, no, I think it took 15 years not being a full-time employee to actually believe this and own this. I think that the only person that can drive change in an organization is a CEO. It doesn’t mean other people can’t influence that change and can’t contribute to that change, but if your CEO is not on board, it’s not gonna happen. This was really hard for me as an employee because I just wanted to change things, right? Like, I thought that this was the right way to do things. And, and even now, like, I surprised some people ’cause they’re like, Hey, my CEO wants me to build this thing. What should I do? Well, clearly you should build that thing. Like, I think we struggle with accepting our organizational realities.

I’m gonna drop one that might annoy a lot of people, a lot of your listeners, but I’m gonna say it anyway ’cause I feel like it needs to be said. How many designers do we know that don’t think they should care about business outcomes? Like, I’m sorry. It’s great that your product is delightful and it should be delightful and that’s good, but if it doesn’t make your business money, you’re not serving the business. We do need to serve the customers. Don’t get me wrong, this is one where I get into deep, horrible arguments with designers about, but like the reality in a for-profit business, the CEO is responsible for growing profit and every single person that works in that organization is responsible for contributing to growing profit.

Everybody. And I think that like we still struggle with this ’cause like we all have our ideals like, and I think design has a lot of great ideals. They also sometimes get in the way of like doing design in a business.

Peter: Mm-hmm.

Teresa: Okay. So that’s the first thing, which, like, still is hard for me to even say out loud because I am such an idealist and I want it to be different.

The competing thing I’ll say is I often tell people you can adopt the habits regardless of how your organization works.

Jesse: Mm-hmm.

Teresa: I think these sound like competing thoughts, but they’re not, because I don’t need the rest of my organization to be outcome focused for me to start to think about outcomes. For me to start to think about the impact of what I might be working.

I might not have access to customers because I might not be allowed to talk to ’em in my organization, but odds are I have some access to them somewhere, whether that’s reading forums online or finding people in my personal network, like with every single habit in the book, there is something you can start with, even if your organization is telling you you can’t.

And so, like, what keeps me sane is holding both of these things at the exact same time. I’m not going to change my organization, but I can change me. It turns out if enough me’s change, the organization changes.

Peter: And you’re tapping into something that I find is part of my work, which is recognizing that nothing happens or very little happens without cross-functional connection and collaboration. There’s this belief that everything you’re doing has to be aligned with all the people around you at all times and needs that support up and down the chain.

And while there’s some truth to that, allowing people, the word that comes to mind is selfishness or self-centeredness, and being, like, kind of to your point, you can do what you think is right. Give that a shot. There’s no one holding you back. Try it out.

Like, you don’t have to always align and connect and integrate with what’s going on around you if you think there’s a different way of working. Now there are political challenges and relationship challenges and you don’t want to just be kind of operating 180 degrees from everyone else around you necessarily, but, providing some permission.

I like what you’re saying because I think more people with the right idea need permission to like, do those things. But I also like your comment about the CEO and that change because yes, you have permission to try some new things, but you also need to recognize the reality in which you’re operating.

You’re not just gonna be able to willy-nilly make change.

Teresa: Yeah. I always tell people, I go, look, you need to rock the boat, but don’t get fired. That’s the reality, right? Like if you wanna work this way, you have to rock the boat a little bit. Just do it within the constraints so that you don’t get fired. And I think that’s these competing ideas of like, I do work in an organization that I don’t control and I barely influence and I probably will never really influence if I’m an IC and I can still have so much agency in what I do all day, every day. Your boss is not looking over your shoulder all day every day and never will be.

So you gotta find what are the safe places that you can start to introduce some of this stuff. And what I love about this is, here’s what I see happen when people do this. They start to do things in weird ways and people get curious about those weird ways and ideas start to spread. And if those ideas happen to work, they spread even more.

And it turns out we do start to influence change just by changing ourselves. And this is why I like William Bridges’ work. We’re all individually changing at our own pace.

Experimenting with AI

Jesse: Speaking of permission to experiment, there’s a lot of experimentation going on in organizations right now, and a lot of folks are not waiting for permission to experiment because they are seeing the opportunity to seize these new technologies of AI to do their jobs and well, in some cases, to do maybe what might be somebody else’s job in the organization too.

And I wonder if you can put on your future forecaster hat. I can see it hanging on the wall there. And speculate with us a little bit about what this technology potentially means for the future of the product trio, future of product discovery, the future of product delivery.

Teresa: Yeah. Okay. I recently interviewed 17 people about how they used Lovable in their work. I will be releasing a blog post about this in August. I also am building two of my own AI products at a depth I did not think I would be at. Like, I just started tinkering and it ran away from me. And now I have a pretty robust AI product.

And both of those experiences have really pushed a few themes for me that I also will be blogging about in August. But I’ll give everybody a sneak peek. I think one of the biggest impacts generative AI is gonna have on how we build is we’re gonna see an even bigger blurring of our roles, and we’re gonna have to get comfortable with it.

And I say this for a couple reasons. I mean, I just talked to 17 people about lovable, most of those 17 people were product managers who were building features that their engineers are shipping without designers. Now, do I want that world? Absolutely not. I want designers involved. But why are these teams doing this?

Because their company hasn’t hired enough designers. They’re still shared across teams. They’re limited by resources and the product managers just trying to get something out there,

Jesse: Mm-hmm.

Teresa: right?

What does that mean for design? I don’t know, but it kind of scares the crap out of me. I mean, most product managers are not equipped and Lovable does a pretty good job.

Like if you just want a basic boring design, it does a pretty good job. But for those of us that want delightful products in the world, it’s not gonna get us there. But here’s how I think about this. There’s always been tools that allow us to build mediocre things, and there’s always been companies that choose to build great things. I think AI is gonna make a lot of mediocre things. So for all those companies that are letting their product managers use lovable to do design instead of designers, they probably weren’t hiring designers anyway. Well, there may be more of them. Yes. But I think we’re gonna see with a lot of mediocre products, design is gonna come right back in importance because we’re gonna try to not be mediocre.

Just yesterday I saw a post on LinkedIn that was like, Lean Startup is dead. It’s free to ship now. We don’t have to learn anything. We can just ship things and here’s what… yeah, since people only hear this, I’m gonna point out that Peter is like passing out over here. Here’s what comes to mind.

Have you two seen the Simpsons episode where Homer Simpson gets to design a car?

Peter: Oh yeah, the Homer.

Teresa: This is the most relevant like pop culture reference of all time right now. What happens when we let anybody design anything? There’s no coherence. It’s a nightmare. It’s not usable, it’s certainly not a Lovable product.

Okay, so that’s the first thing I think is gonna happen is we’re gonna see a lot of blurring of rules. We’re gonna make a lot of mistakes. We’re gonna think some rules aren’t needed. We’re gonna see people try to ship products without engineers because delivery’s free. I I mean, come on, is it really free? Is it ever really gonna be free?

We’re gonna see people try to do the same without design. Some engineers are already talking about how they don’t need product managers anymore because ChatGPT can just write a PRD for them. I think we’re gonna see that all three of these roles are critically important and they’re gonna stay critically important.

The second thing I’m seeing: Getting quality out of LLM based products and features is really hard. So I’ve been talking to a lot of people about how they’re building LLM features and products, and here’s the mantra here, over and over again, getting to a prototype that looks like it works is really easy.

You can do it in one or two hours. Getting something that reliably works at scale for thousands, if not tens of thousands, if not hundreds of thousands, if not millions of users is really hard, and we’re just starting to learn how to do it. And the ways that we’re learning how to do it require so much cross-functional collaboration to do it well that I think we’re gonna be forced to work even better together.

So this is what I’ve experienced with my own product. Like I started to get into the world of AI evals. It’s a big, messy topic that most people have not even heard of yet, but it’s basically evals answer the question of how do I know if my LLM feature is good, right?

It’s non-deterministic. I’m getting all kinds of outputs. How do I know across my production system that it’s good? There’s lots of ways to do evals, but the best way to do evals requires the domain expertise of your designer and/or product manager, and most importantly, your customer. And engineering expertise to automate it and get a large enough sample to represent what’s happening in production.

Right now, like, I ended up in this community of like AI and ML engineers, ’cause I took a class with some of them and they’re all talking about who has this domain knowledge and I’m, like, your product managers and your designers and your customers, and they’re like, oh, would they do this stuff?

Like, we’re just learning. Nobody knows. It’s like, it’s literally like 1994. Like, we’re just learning. And I think, like, the little that I see so far, it’s gonna require, like, not two in a box, three in a box. I mean two people at the same desk working together, discussing what’s the right way to do this.

And maybe eventually we’ll get tooling that like support handoffs, so we’re not literally sitting at the same desk doing this. But right now, that’s what good looks like.

Peter: How has building your own product changed how you think about building product?

Teresa: I’ve become an engineer. I’ll tell you, I spent my weekend like configuring VS code and using cloud code and setting up a proper dev and prod environment. Like I am pretty much an engineer at this point. Maybe not a very experienced engineer, but an engineer. Why don’t I just hire an engineer?

Because I don’t know how to separate the engineering work from the product and design work yet. Like I don’t even know what that looks like. Partly it’s ’cause I’m still learning, but I’ll tell you, I’m hearing this from other teams. Like the biggest question I’m getting from other teams about AI products is like we’re arguing over who should do what because there’s parts that are not that fun, but they’re critical to the quality of the product, right?

So right now, I personally am trying to do all the parts because it seems this question of who’s eventually gonna do what seems really messy. So I’m trying to understand all the parts.

Peter: You are your own triad.

Teresa: I am my own triad. It will not stay that way forever, that it’s not sustainable. But generative AI does make it possible. I mean, it’s helping me fill the gaps that I didn’t know how to do myself, for sure.

Jesse: And it seems like that’s the opportunity for a lot of folks, whether you sit on the product side, whether you sit on the design side, whether you sit on the engineering side. If you are one of these people who already naturally has a toe or two dipped in these other waters. You know, if you are a design minded product manager, if you are a product minded design person, if you’re an engineering person with either of those sensitivities, the opportunity is here for you to create a different kind of value than you’ve ever created before.

Teresa: I actually think our boundary spanners are gonna be our most valuable employees. I would argue they’ve always been our most valuable employees. Now, I have a bias. I am a boundary spanner, but I do think when we can speak to multiple roles in our organization, it does increase our value. And I think that’s gonna get exacerbated by gen AI.

Jesse: Teresa, what are you most looking forward to with what’s to come for design and product?

Teresa: I’m actually really excited about what generative AI unlocks, so like I personally have been nerding out on and like the tools that I’m building are all around how do we teach skills? Use generative AI to teach skills and help humans build skill. So like one headline I see a lot, like the founder of Anthropic gave a talk where he said all entry level jobs are going away.

And this like scared everybody, like how are we gonna develop talent? And I actually, I’m not so sure I agree with him because I think we’re gonna be able to level up entry level employees a lot faster because of generative AI. I think people are gonna be able to enter their careers with a lot more experience.

Like in my head, in the back of my head, I have this like Neo in the Matrix, like just downloading a skill. It’s not quite gonna be like that, but I think we do like now have the ability to let people build hours of practice on their own time through interaction with LLMs. I think it’s gonna require that we get really good at designing what that practice looks like. I think like apprenticeship is gonna look really different. And I’m actually really excited about that. And I’m someone who loves to mentor and to teach, but I’ll give you an example. The first tool I built, is I have a customer interview coach, so my students in my interviewing class submit their transcript and they get very detailed feedback on how well they conducted a customer interview and like way more detailed feedback than any of my instructors would ever have time to give them.

And they get this every single time they submit a transcript. It’s really unlocking deliberate practice. This is like version one. I can tailor this to like, oh, it looks like your last three transcripts, you’ve struggled with this particular skill. So you’re only gonna get feedback on that particular skill ’til you get better at it, right?

Like it unlocks personal learning journeys. And for me, what I love about it is now I have visibility into learning outcomes. Like as an instructor, I can go and observe, but now I get concrete data on every single student in my class. This is magical.

Peter: Hmm. Hmm.

Teresa: Now, it’s not like I gave you a certificate because you came to five class sessions.

Now, it’s I gave you a certificate because I saw your interviewing practice evolve and you reached a level of competency and it was all done through generative AI. Pretty cool.

Jesse: I love the idea that these technologies are not just going to replace entry level people, but actually enable people to level up from the entry level more quickly. And the idea that these technologies are not gonna replace anything human so much as they are going to augment and amplify the human abilities that we already bring to this work.

Teresa Torres, thank you so much for being with us.

Teresa: Thanks for having me.

Peter: Yes. Thank you so much for joining us. I know you’re pretty easy to find out there, but, where do you point people to so that they can follow up with you?

Teresa: Yeah, so most of my stuff is linked from producttalk.org. I blog every Wednesday. I might be adding a Monday AI themed post starting in September. That’s a little sneak peek also of what’s coming. And the book is Continuous Discovery Habits, available worldwide.

Jesse: Thank you so much.

Peter: Thank you.

Jesse: For more Finding Our Way, visit findingourway.design for past episodes and transcripts, or follow the show on LinkedIn. Visit petermerholz.com to find Peter’s newsletter, The Merholz Agenda, as well as Design Org Dimensions featuring his latest thinking and the actual tools he uses with clients.

For more about my leadership coaching and strategy consulting. Including my free one hour consultation, visit jessejamesgarrett.com. If you’ve found value in something you’ve heard today, we hope you’ll pass this episode along to someone else who can use it. Thanks for everything you do for others, and thanks so much for listening.