Human Judgment vs. Machine Output

Show Notes

About the episode:

Dan Saffer—designer, author, and CMU professor—joins Peter and Jesse to take stock of design education at a moment when everything is in flux. What’s worth teaching when the tools change weekly? How do you define quality when AI can spin up a passable prototype in minutes? And what are today’s students teaching their professors about where design is headed?

Special Announcement:
Jesse and Peter are excited to announce the launch of the Intentional Design Leadership Circle, a six week cohort for director and above design leaders who are ready to move beyond surviving and toward leading with purpose. Each week we’ll facilitate discussion on a different facet of leadership, and by the end you’ll have new strategies, frameworks, tools, and a community of peers who understand exactly what you’re navigating.
Find out more and register.

Transcript

Jesse: I’m Jesse James Garrett,

Peter: and I’m Peter Merholz.

Jesse: And we’re finding our way,

Peter: navigating the opportunities

Jesse: and challenges

Peter: of design and design leadership.

Jesse: On today’s show, our Adaptive Path colleague Dan Saffer joins us. He’ll share some of what he’s learned on his journey from design book author to Silicon Valley design leader to his current work training the next generation of designers as a professor at Carnegie Mellon University. We’ll hear his perspective on what it takes to prepare the next generation for the age of AI, as well as what they’re teaching him about design process, design’s value, and where that value is headed.

Peter: Dan, thanks for joining us.

Dan: Hello. Thanks for having me. this is fun. Unusual seeing you all so far away.

Jesse: It feels so natural, doesn’t it?

Peter: For those listening, we have with us Dan Saffer. Jesse and I have known Dan for over 20 years now. I believe I met you, Dan in 2003 or 4 at a Designing Interactive Systems conference where you were talking about metaphor in design, and you were getting your master’s at CMU, the very program similar to what you’re now teaching.

We’ll get into that in a moment. You worked with us at Adaptive Path for a number of years, and then you went off and had your own journey including through Silicon Valley, and now have found yourself… are you a professor at CMU? What is Dan Saffer doing now?

Dan: My official title is Assistant Professor of the Practice at the Human Computer Interaction Institute at Carnegie Mellon University. Say that three times fast.

And what that means is I worked a lot in the industry and now come back and teach students how to go be industry professionals mostly in design.

I mean, I teach design in the school of computer science. I teach people that want to be UX designers, product designers, AI designers, product managers, user researchers, kind of a wide gamut of folks that come through the program.

And that could be anything from our PhDs to master students and people that are doing stuff at the business school, like people that wanna be PMs or people that are just interested in design and want to come find out what that’s all about, and data scientists, and really, you name it, like we are one of the most popular undergrad, second majors in the entire university.

So it’s really fun. A pretty wide variety of people come through, some of whom it’s, like, this is what design is, and some of whom have worked, you know, in the industry for years.

So it’s been a very interesting three years to convert from practitioner, design lead to design professor. And yeah, speaking of a journey and speaking of finding our way.

Peter: Well, we’re gonna get into that, but you said the phrase, I teach design. What does that actually mean? What does it mean to teach design in this HCI program?

Teaching Design at a University

Dan: Wow, this is a super long topic. So HCI has got a number of different disciplines that are a part of it. So there’s computer science, there’s psychology and sociology, there’s learning sciences. There’s engineering and there’s probably one I’m missing. But there’s also design.

So, in the design faculty, is kind of where my subclass is, is all about teaching a number of different things. We teach service design. We teach interaction design fundamentals. We teach advanced interaction design. I teach a lot of design and AI and we also teach some kind of entrepreneurship, like getting a startup out and those kinds of things.

So that’s kind of what teaching design is currently. So I do a lot of the kind of advanced and AI type stuff, although I have taught pretty much everything else that I just mentioned at one point or another.

Jesse: So in addition to the work that you’ve done over the years, you’ve also written a bunch of books, right, where you have explored different facets of practice within design, interaction design, digital product design. And I’m curious about the threads that you’ve seen that are influencing the way that you engage with students now around those fundamentals.

Like, you wrote one of the classic books on interaction design, Designing for Interaction. You were talking about gestural interfaces before other people were, you were talking about microinteractions before other people were. I’m curious about how all of that work around establishing the deep patterns of how humans interact with these systems is playing out now as we’re seeing, really, this wave of radical change as you are engaged with that leading edge and these AI technologies and how they fit in.

Dan: How I like to think about it is that there’s really only two things right now that are really worth teaching.

One is the fundamentals, and one is the stuff that’s kind of on the cutting edge.

And the stuff in between is really kind of up influx, like the design process is right now kind of in flux. And different methods are in flux. User research is in flux. There’s lots of stuff that is kind of in flux right now.

But there are certain things that never change because human beings never change. So there are certain rules and principles and foundational things like typography and what looks good and how things act, and getting a product sense and reframing and problem finding and meaning making. All those things that are kind of the core, basic, super-skills of design, I think are a really important thread to maintain that connection to Henry Dreyfus and George Nelson and all the people, you know, all the designers that came before us many, many years ago, but now we also have this, like all this new stuff that’s coming and that is just something that, yeah, we are trying to kind of keep up with, figure out what is worth teaching.

So that students can use that to generate things, to generate concepts, generate ideas, to help them you know, do kind of the old school production work. But what we’re trying to keep is the judgment and the thinking part of it. The solving the right problem, the strategy part of it.

So trying to keep those things that are really important about design, while still saying, okay, there’s this new technology. We’re all fumbling around with it. It’s clearly nascent, it’s clearly important, and what do we do with it? And so those two things are the real pillars of what I’m trying to teach and imbue students with my pearls of wisdom these days.

I mean, it’s funny ’cause, like, the original books were meant to be… the Designing for Interaction book was meant to be a textbook. It was meant to be a standard book for design students because there just wasn’t anything out there like it, you were cobbling together all these different articles.

And so the idea was, let’s put it together as a textbook that can be taught. And I’m assuming, because I still get like a dollar or two a month in royalties from it, that somewhere somebody is still teaching it.

Peter: Since you just mentioned the book and maybe whether it’s Designing for Interaction or Microinteractions, what, given what you are working on now, and what you’re witnessing and experiencing, what do you look back at what you have written and think, oh, that no longer holds, that was somehow inaccurate or inappropriate, or you were able to successfully kind of establish some foundation there that does still feel relevant today.

What’s Relevant, What’s Not

Dan: Some things you really have to question, like, is it worthwhile to be doing wireframes now? I mean I spent a lot of time in the book talking about wireframes, and we’ve kind of talked about them for years as being a great way to get people to look at your designs without focusing on the visuals,Right? That’s the whole purpose of wireframes.

Jesse: Right.

Dan: Is that valid now when I can spin up a whole, basically a whole branch, a whole prototype of something that kind of works or looks and kind of feels like… Should I bother teaching wireframes now?

I do, because of exactly what I said,you know, I’m trying to give students a lot of things that are in their toolbox that they can pull out and use when it’s appropriate. That’s the hope where they’re like, oh, wait, if, people keep getting stuck on the visuals here, maybe we should just do the wireframe version so that they can go back and kind of backfill or, pull this dusty old tool out of their toolkit and blow the dust off of it and be like, here, let’s try this old thing and see if it still works.

There are definitely things that I have found that are becoming more relevant. Things like task analysis now is incredibly important when it comes to thinking about agents, for example.

Before you remove an entire person’s workflow or change their workflow by adding in an AI agent to it, you should probably understand that workflow and all the decision points and what’s happening and what the different nuances of that workflow are before you just plow through it with an AI agent and push the button and have it try to do everything at once.

So there’s lots of stuff like that. I mean, remember when prototyping was incredibly hard? it was almost impossible to get something that looked and felt like it worked. I mean, but that’s the olden days now. Like now in an afternoon you can get something that looks and works pretty well.

Jesse: So what are the implications of that for designers and design as an activity?

Dan: I mean the, biggest implication of that is I do no thinking upfront, and I just am like, type, type, type, type, type, prompt, prompt, prompt, go, oh, look, here’s the app. It’s finished, it’s done. Here’s the prototype.

That’s the big drawback to it, right? Where you are quickly prompting and then you are not reviewing to see if it’s good, if it’s valuable, if it actually works the way that you think it should.

Now the bonus to it is that you can spin up four or five of these in the time that it used to take to do a piece of one, and you can then test them all and play with them all and try them all out and see which one really works.

That’s the holy grail of this whole thing is that you can have these things that look and work really well and you can test them and you can get them in a week or less.

Jesse: I think it’s interesting and exciting to think about this as part of an individual designer’s creative process in fine tuning because it seems to me like, why would you need a robust, fully built out, fully detailed, fully realized prototype?

Hopefully it’s because you’re asking some pretty fine-grained questions across those different prototypes about what is the most valuable design process.

What I wonder about is how this plays out in a cross-functional environment, where the designer isn’t the only one coming to the meeting with a prototype. And in fact, it starts to look a little bit like High Noon, like, you know, show down at the O.K. Corral, where it’s like, I got my prototype. You got your prototype, let’s shoot it out, and let’s see whose prototype takes the day.

And I wonder about the implications of these tools for collaboration, not just for individual ideation.

Dan: Yeah, absolutely. I don’t think this is also anything that’s super new. I remember way back in my career that PMs would show up with screenshots of what they wanted to be designed, and it’s a similar kind of thing. Now though, it just looks great. I mean, it looks real in a way that a marked up screenshot for me to design never did.

And I agree. There is this weird, now, tension kind of between PM, design, engineering. It’s like, how do we work on these things together and make sure the right idea wins?

That I think is what is really important. And that is also the thing that AI does really badly, again, which is, are you solving the right problem? Are you going about the right way? Are you framing the problem correctly?

That is something that AI does not do well. If you come with your prototype and it is for this problem, and I come to it with a better framing, then, probably, hopefully my idea will win out because it’s like, oh, I see how you thought about this. This is a different, a better take on this, ’cause it’s solving the real problem, not the problem that we thought we had.

Our classic old consulting problem. You come to us with something that you think is the problem. We find out what the real problem is.

Jesse: Right.

Peter: Yeah, RFPs often had the solution inferred, and we would have to try to back out of that and probe that.

Dan: But I have been talking to a lot of industry leaders, many of whom don’t want me to talk publicly about kind of what they’re doing, but that’s fine. I’m gonna genericize it. But a lot of it is that there is really this merging and blending of the teams, where a lot of the time that designers used to spend in their silo making mocks and prototypes is now spent alongside the PM and the engineer working to refine a prototype or multiple prototypes together.

And, to me, that’s a pretty great solution. I think that to have that kind of close collaboration where you’re working that closely rather than often doing kind of waterfall kind of stuff, where then you toss it over the wall to the engineer.

But again, you have to make sure, just like with our old friend Agile, that you have the time upfront to actually think about what is the right problem? Are we doing this the right way? Are we solving the right things? Is this worth our time to actually spend, to refine?

And I’ve also heard that on certain teams, there are definitely designers who are now shipping production code because it is in places where there’s not enough engineering talent to really handle those, or it is stuff that engineer’s don’t have time for, or don’t have much interest in. So like, final, like, production polish. A lot of times the designers are now doing that and I love that.

I was like, ding, ding, ding, ding, you know, like, ’cause you used to have to file ’em as like a bug report or something like that. Trying to get people to actually work on some of these polish things and being able to go into the code and do little tweaks and ship. Wow. Amazing.

Working in Design Teams

Peter: When you were talking about teams, I understood it to be a designer working with cross-functional product manager and engineer. Something I wonder about is designers working in design teams.

When we collaborated at Adaptive Path 20 years ago, all of our projects were done by teams of designers that had complimentary skill sets, you know, range of experience and it wasn’t just us. Most design agencies would staff teams of designers because that led to better design when you have more brains tackling a problem.

Over the last however many years, 10, 15 years as design’s moved in-house, you tend to get the lone designer, even pre-AI, right? The lone designer working on a Scrum team, in their silo trying to make stuff.

Part of the reason I co-wrote the org design book was to see if there was a way to team-ify design in-house, ’cause, again, I believe designers not just work best, but do their best work, like deliver best when they can work with one another.

I’m curious what you’re seeing either in your observations in industry or college contexts. Usually you have the students working together on projects, and so there’s a teaming there. How does this supplant, support, change the dynamic of, collaboration within a design team when adopting these tools?

Dan: There are two things that I have heard about this.

One is that in a lot of these organizations, there’s still a design organization and then teams are still formed, like they have been for the last you know, however many years.

Someone gets lent out to the onboarding team or the safety team or the transaction team or whatever it is, but they still live in the big design org. I don’t think that that particularly has shifted very much.

And because it seems so easy for one person to be able to spin up these things, the concept of a team is completely lost. That you have people that are doing everything and there’s no one over their shoulder, there’s no one alongside them who is doing the plusing work, as we used to call it, at Adaptive Path, making things better. There’s no one doing that.

And the AI certainly isn’t gonna tell you, Hey, this is a dumb idea. You shouldn’t be doing this. So I think some of that diversity of opinion and diversity of ideas is going to get lost.

And I do worry about that as we start to move to these single person being able to do the PM role, the engineering role, and the design role all at once in one thing, is it just going to be everyone doing everything poorly?

So that’s what I do worry about.

Jesse: It is an interesting question because it’s almost like, you know, isn’t this where we came in?

Dan: For sure. Yes, time is a flat circle here.

Jesse: Yeah, exactly. When we entered this industry in the nineties, what you had were software developers or maybe software engineers who were expected to manage their own projects. They were expected to conceptualize user experiences. They were expected to make the technical, architectural decisions about the components and the modules and the technologies and all the bits and bobs that would need to go together to create that experience.

And then they would be expected to write the code that put all that stuff together. And that was the job for decades before the advent of user experience design as a discreet discipline within the product development process. And, you know, along the way we got product managers and we got, you know, growth hackers and all the other kinds of variants that we got.

Peter: Barnacles.

Jesse: Yeah. Well, but that’s the thing is, like, if it’s all kind of coming back to this idea of the creative contributor who can think on multiple levels at once, what does that mean for the mandate of a design team within an organization? And what does that suggest about the actions that a design leader needs to take in order to activate and realize whatever that mandate turns out to be?

Dan: Now I have heard that in these organizations that I’ve spoken with, and they are all fairly large, have pretty good sized teams in them, that they’re still doing things like design crits. So that is why they still live in a design department. They do have like weekly crits and those kinds of things that are still part of design practice.

And I think that is good because if you are a solo designer working on a single team with a PM and an engineer and maybe a data scientist, you’re not getting that kind of design critique that you might get in larger organizations.

But at a smaller company or a startup, are you gonna get any of that? Probably not. You’re probably just going to be expected to do it all as yourself. And frankly, the tools are not ready there yet. And most of the people that I see get sucked into startups, particularly young designers, they’re not ready for that kind of overall responsibility yet. They don’t understand it.

And so it is, it’s a concern. Now that being said, I know and have worked with people that could rock this, could really like just take this tool and do, like, amazing things with it on their own because they have a great product sense. They have a little bit of a coding background. They’re methodical, they have really good ideas.

They actually bother to think through problems. They do thinking upfront.

So there are a certain set of people that I think this will really, really ignite. But I don’t think that’s everybody.

Jesse: Mm-hmm.

Peter: I’m curious in the context of your classroom, I’m assuming students are doing projects, and I’m assuming they’re doing them in teams. How are you seeing your students navigate collaboration with these tools, that can so much encourage everyone kind of doing their own thing, right?

These are not collaborative tools, usually. Figma is, but the various LLMs are very much like me and the machine are having a dialogue and getting anyone else involved is awkward.

And I’m wondering what dynamic is emerging that you’ve witnessed, if any?

Dan: It depends on the class. So in the advanced interaction design class, it is very kinda one-on-one. But we do have crits with the other designers in the class because that is a, traditional design studio class.

In the designing AI products and services class, we deliberately set up the teams in that class to have technical people, business people, and designers on them.

And so we deliberately try to make a simulacrum of what it’s actually going to be like to work with people of other disciplines. And that is always one of the biggest highlights of the class.

And one of the things that the people love the most about the class, because they come in and they’re like, especially if they’re coming from like the design school or the engineering school or the business school, they’re like, I have been trained to look at things through this very narrow lens. And having to actually discuss and debate even when they are using the tools to put together an agent or a gen AI product or those kinds of things, they are all debating and talking about it as a group of people with different backgrounds.

That’s kind of the best we can do, because you’re right, the tools are not currently set up to do collaborative work.

Peter: Does someone end up, like, owning the keyboard and the other two are talking to them…

Jesse: Whoever controls the prompt controls..

Dan: Yes. I, I, think it does end up like that honestly, I think it is like…

Jesse: the product.

Dan: There is someone who is the designated AI lead when they are prototyping, and that is the person who is in charge of that piece of it.

Now, I will say that before anyone starts to prompt, we force them to spend a week or two trying to figure out what exactly it is that they’re going to build. We don’t let them just run off and just start prototyping. We really force them into doing some ideation, some ranking, some consequence scanning, thinking about what could go wrong. Thinking about things like the business, like the financials. Like, is this gonna be actually valuable to the business? And then, like, do people want this? Is it actually a real need that people have?

So we really force them to go back to design process. We kind of force them into like doing the upfront stuff before they ever touch the keyboard.

And I think that is a great practice for them to have going out in the world.

Jesse: Do you think they see the payoff, the benefit of that?

Dan: I do actually. I think a lot of them really do think that it is valuable, that they just haven’t wasted their time building prototypes that no one wants or that no one will pay for or no one will buy.

And we especially get a lot of designers who are like, oh, I’ve never thought about the business side of things before. I’ve never thought about how much this might cost. And that’s great.

And then we’ve get a lot of engineers who are like, well, I’ve never thought about whether somebody wants this or not. So, it is eye-opening from a lot of different sides, and I think that they do find value in it.

Now, whether they are continuing this process out in the real world, I’m not sure. We lose them after they launch, but I think it is a valuable practice for them to have in their lives.

And certainly in the advanced class, we definitely spend a lot of time talking to them. Like we give them a product brief that sucks and have them try to figure out why it sucks, where it sucks, how to fix it, how to reframe it, all of that before ever again doing any kind of concepting or prototyping or those kinds of things.

Although, I will say sometimes we do have them use the given crappy brief to make the prototype of what is being suggested and see how terrible it is and be, like, is this, what, is this real? Is this really what you want?

Jesse: Right. Right,

Dan: So that alone is an actual valuable tool ’cause you can spin up a really crappy prototype based on really crappy requirements really fast and see immediately like, Ooh, I can see some real problems here. It really blows holes in the PRD, which is great.

Is “Taste” a Skill?

Jesse: Yeah. I love the idea that the reason you’re getting crappy results is because you haven’t been specific enough about what you’re asking for. Or there are inherent contradictions in what you’re asking for that have to ultimately be resolved.

Which kind of comes back to what you were saying before about what design actually brings to the equation, which is problem framing and this kind of taking problem framing as a process, really to shoot a frame over at the machine and see what the machine comes back with and say, okay, is this actually what we wanted? And creating space for that human judgment in the process itself.

And I think that the fear, and for some people, the promise, is that that first one that comes back is gonna be exactly the right thing. And I wonder about not just the process implications, but the mindset implications for for designers, other creative people involved in this creative process who have to be able to look at what came back and actually say, you know what, this isn’t good enough. This isn’t ready for production. This isn’t what we wanted.

And whether there’s a skill set there. I mean, you referenced taste, and I wonder about where that fits into how we frame the value that the human on the human side of the keyboard brings to that interaction.

Dan: I think that we should think about taste in a broad sense. I think there is a lot of design copium happening right now. Where it’s like, well, my brilliant aesthetic judgment will save me from the AI machine, and I don’t think that that is true.

I think these things are gonna be able to present pretty good looking things in short order soon.

And I do think that there is actually a lot of underlying taste about UX that I think is getting lost here. Like, does the flow make sense? Does the information architecture make sense? Does the problem make sense? Are we making it too difficult to do these things? Are we making users do too much work? Is this wrong for our particular context? Can we do this thing because we have compliance issues or regulations that we have to follow?

So I think there’s a lot of taste questions that are really like a lot of judgment questions that have to fall into the big taste category. I think it’s taste broadly in terms of both look and feel. Does this work for our users, our organization, our context, and I think that is really important.

Now, I will say, caveat, that there are times when that push the button, the first thing that comes out is gonna be fine, is gonna be fine for certain basic tasks.

Jesse: Right,

Dan: If I’m making a thing for myself and, I’m the only one who has to see it, well, I don’t care about things like accessibility, does it make sense to a broader audience? Does it make sense at scale? No, it doesn’t. It just has to work for me. And so something that the machine cranks out that’s 80% fine is fine and I’ll just use it.

Now, that just does not work when you’re talking about working at scale, working for lots of other people. People still want good designed things and will always, I think, want good designed things.

But there are these situations where good enough is good enough and we see this all the time with, you know, just to take an old school example… My restaurant site does not have to be the most cutting edge website in the world. It just has to be clear and see things. And a template is perfectly fine for that.

And I think we will see the same thing with AI, that it’ll create a little app for us. That’s fine. It’s based on best practices and a kind of perfectly okay looking generic user experience and UI and it’ll be fine.

Defining Quality

Peter: You’re mentioning concepts of good and good enough, which are, part of a definition of quality.

Dan: Mm-hmm.

Peter: And I’m curious how that conversation is going on in school or with some of these companies you’re talking to. I don’t recall if you were at Adaptive Path when I tried to define quality inside the company, and caused some… you do, okay.

Because it caused some consternation because where I landed was that quality was defined through a conversation with the client. Like it was an agreement, was quality. And if the client was happy, then the design was good or good enough.

And there were people internally who were like, no, ’cause our clients aren’t gonna have the same standards that we do and we should be aspiring for something greater. Fair.

But when then pressed on how do you know when something’s better, it was all, you know it when you see it, which to me doesn’t hold water, right?

And so I’m wondering, as you’re talking about taste, and I know you’ve talked about discernment and you’re in a design school where the goal is to help people develop their craft, and the only reason to develop craft is to make things better, right? Is to deliver higher quality stuff.

How does that quality conversation happen? Or what is that conversation around quality that you’re taking part in or leading with, whether it’s students or, again, the kind of relationships you’re having outside of school.

Dan: It is definitely a very hard thing to define and try to wrap your head around, particularly for students because it’s like, well, this is good, good enough. And I actually last semester, I came up with like a metric or a rubric for myself, which was that this is better than it needs to be.

And for me, that is, that is the mark of quality. Like it didn’t have to be this good, but goddamn is this good? This makes doing this, whatever, this thing is so much better.

Jesse: That is literally word for word what George Lucas said about The Empire Strikes Back. He was disappointed in the production team because it was better than it needed to be. They spent so much of his money on it.

Dan: Yeah, I mean, to me it’s, like, it is that deep understanding of the goals, the task, and solving that underlying problem that I think really sets something apart as being high quality. And again, it’s really better than it needs to be. And that is a surprisingly hard concept to convey to students.

Peter: Well, yeah. How do you teach that? Or, you mentioned crits that you say your advanced program is part of. Like, how are you helping them understand that perhaps what they are showing is not good enough and that it can be better? What does better mean? Is it simply better because it solves the problem better that we have defined, or are there standards of quality that you are trying to help them appreciate and understand? Kind of more common standards of quality.

Dan: Yeah, some of this is a lot of the kind of tacit understanding that you get when you’re doing the kind of one-on-one crit, I guess there’s a number of different things that I’m looking for and trying to get them to see. And some of what I think I’m trying to teach is getting them to see their product through my eyes, how I’m seeing it, and the things that I think are off.

Not that my level of quality is insane, but getting them to notice the things that I do that they don’t. I remember two years ago I was like, that thing is off by one pixel. And the student was like, how did you see that? And I’m like, well, I’ve been doing this a long time. Like I can tell when something’s not aligned, even, you know, even a little tiny bit. It feels off, it feels like it is not quality.

Now, something like that I would not even consider. That’s not quality. That’s like baseline craft, like make things align.

But you would be astounded in the fundamentals class, how many people are just like, what do you mean grid? We don’t need a grid.

Jesse: Well, I think it’s interesting how this applies to where AI tooling comes into play, because it comes back to that same thing. Can you tell that it’s off by a pixel or not? And can you tell that the results that you’re getting are off in some way or not? And it’s interesting the way that what you describe is the honing of awareness and the sharpening of one’s ability to detect the stuff that you wouldn’t have detected early in your career.

And to detect the misfits.

Dan: Right. ‘Cause so much of it is like, it’s all nuance, it’s all small stuff. And it is just getting them to be like, well, why is this three steps instead of two? Can you put these two things together? Or you know, are you making users do too much work here? That’s a huge one for me right now, where it’s like, well, we’ll just have users when they get to a spot, they’ll log in and they’ll do all this other stuff for free.

And I’m like, no, they will not, they will not do that for free unless there is a big payoff for them, unless there’s this value exchange from doing all this work. And trying to get them to see, like, don’t make the users do unnecessary work.

Trying to figure out ways to streamline the flows or make things that are invisible clearer. Like, you know, how the rules work. So one of the problems that we have them work on, that they just turned in, is, like, working with a dating app, trying to figure out how to add this new feature to a dating app and then they have to start thinking about, well, how does this work for people that have paid for the dating app and people that have not paid for the dating app?

And trying to figure out what the rules for that kind of system is, is pretty challenging in some of their cases. And that would be something that would be really hard to try to describe and put into an AI right now.

You can do it, it’s just, there’s a lot of complex thinking that has to go into that and trying to get them to, A, think about it in the first place, and then B, figure out what those rules should be. And then C, how do those rules manifest in the system and where do they manifest is a big kind of journey that they go through.

But yeah, teaching quality is extremely hard. And especially, you know, you get a semester, you get three or four months to do this. And I always am like, well, I hope that wherever you land after this will be your finishing school. Where you will have, I mean, just like I did at Adaptive Path and trying to finish kind of like what you’re starting the kind of education journey through an apprenticeship

Peter: Let’s talk about that.

Dan: Finding our way,

TM.

The Plight of the Junior Designer

Peter: Almost a month ago I did a guest lecture at a grad school course at Berkeley’s I School. And I don’t get a lot of opportunity to engage with the youths. And I was kind of taken by the challenges that they foresee as they consider that next step.

I’m wondering what you’re seeing, what you’ve seen from your graduates and how they’re faring, what you’re hearing from your students or hearing from those graduates as well in terms of, right now there’s a lot of commentary, you know, there’s that Figma report that said that hiring managers are even less likely to hire juniors now than perhaps they have been.

And I’m just wondering kind of how that’s playing out in your context.

Dan: Yeah, I mean, it used to be that most of our grads would be hired before they left school. Like, and that doesn’t happen now. I’d say maybe, 25%. I’m actually just kinda guessing here. You know that it’s a small portion of them that actually have jobs right as they leave.

Peter: I remember, when things were nuts, that there were… a significant portion of students at CMU had a job for them before they started school. The fact that they got accepted into CMU meant that the Googles of the world were like, go to school, but know that, you know, you have an offer from us.

Maybe not a job, but at least an offer when you graduate in two years.

Dan: Yeah, that’s exactly right. they would come in fall and interview you and give you a hire Before you graduated and then they just hoover you in when you were done, which is incredible. And that stuff never happens now, except for maybe some really technical roles. Obviously if you were, an AI computer science person right now,

Peter: Sure.

Dan: That…

Peter: But it’s not happening out of design anymore though.

Dan: It is not happening out of design anymore. Now I will say in the three years that I’ve been here, it has definitely gotten better. It was pretty bleak the first two years that I got here where it just took students a very long time just to land on their feet, even with a lot of help and support and trying to connect people.

And we do a lot of things like portfolio reviews and I spend, a lot of my time in spring and summer looking at portfolios, trying to help get people ready for school, get ready for post-school, get them kind of launched.

And I’m actually seeing that yes, there is a slight shift that I think is happening and I think that there are people that are looking for new grads, but you gotta be very AI forward, that you have to have those skills to be hired because those are things that a lot of the in-house people don’t have…

Jesse: right.

Dan: …or they are things that at places like Anthropic or you know, at at OpenAI or you know, or Google or any of these kind of big companies that that’s how they work now.

And so you have to have those skills to get in. And so that is weeding out a lot of people on the job market right now who have years of experience. They just don’t have the AI skills. In order to fit into their new process or where they want to or where they wanna be, or they want to get people in who can help teach these new AI skills to their existing…

Jesse: Right.

Dan: …staff.

Jesse: right.

Dan: see glimmers of hope in the hiring world.

Jesse: It is interesting to think about what this means for design leaders. So, you know, we’re closing in on the end of the first quarter. I’m a design leader and I’ve got a certain amount of budget that’s been reconciled. I can can hire four juniors, or I can hire two seniors or, you know, I can create some kind of mix there and trying to figure out how I craft new roles on my team and how do I shape my team for the future.

On the one hand, it feels like the value proposition of design itself is pulling away from execution. It’s pulling away from pixel level delivery, right? It is much more pulling toward the problem framing skillset that you talked about. The kind of thing I would look to seniors to do mostly, but I can’t have a team of all seniors.

For one thing, they’re too expensive, and for the other thing, there aren’t enough of them. So I need to be able to build out my team with a group of people who have a broader skillset. But I can’t look to juniors to do what they used to do for me, which was production and execution. So it almost suggests that there is a reframing of the ladder within design, so to speak, because juniors don’t prove themselves in the same ways.

You don’t prove yourself by your mastery of a tool. You prove yourself by mastery of a system or mastery of a problem space potentially.

Dan: I will say that I do think that juniors are not in Figma pixel pushing as much. Although I still think, for the foreseeable future, Figma will be a tool that people keep using and will continue to use. And I think you’ll be moving a lot of stuff from Figma to code and back again, and eventually there’ll be some kind of merger there.

I do think, however, that juniors will do some of that production work. I think the difference is that that production work will then actually go to production…

Jesse: hmm.

Dan: …will go, out and launch. I think you’ll have juniors who are able to make tweaks and bug fixes for small things.

And that’s how they’re gonna cut their teeth and launch it and see what happens. And that’s what they’re gonna be doing.

Peter: It’s almost… I started doing a lot of qa. It’s almost like, back in the day when, a good way to understand how software worked was to file bug fixes and just to understand kind of those nitty gritty details.

Dan: Right. And that’s how you’ll start to understand the design system. That’s how you’ll start to understand the customer base. They will be put on these smaller things or they will be part of a larger team with more senior people to do some of that kind of work.

Now they probably won’t be doing as much Figma e stuff. They’ll be doing a lot more building kind of stuff. So I think if you are a design leader right now, I think starting to get people in and starting to see how this generation works with things, I think even having one or two of them on your team is gonna be really valuable.

Not that I am a hundred percent biased and I will, and I will be, I mean, it is sometimes shocking to me how this works because I’m like, what do you mean you built an agent to do this thing and you built your own design tool to do this little thing?

Like that stuff to me is, like, mind blowing. I’m like, wait, you’re able to do that? And so that stuff is really super cool and super interesting.

What Are “AI Skills”?

Peter: Earlier you used the phrase “AI skills” and I think you just explained a little bit of what you mean by AI skills. And my org design brain kind of stood up because I think a lot when it comes to how we define roles, right, those roles are often built on skills.

So a product designer will have interaction design skills and visual design skills and information architecture skills. And those skills are basically tool agnostic, right? We are using the same skills for the last 20 years. We just, first, we used ’em in Photoshop or in Visio or in OmniGraffle, and then we migrated to Sketch or Figma and et cetera.

You said AI skills, and I’m wondering what you mean by that. Like what does it mean to develop skill nimbleness, familiarity, comfort, competency with AI in a way that is maybe analogous to developing skills with interaction design or information architecture?

Dan: I think there’s two ways to look at it. One is familiarity with the technology itself. Understanding what AI is capable of, understanding what is required to make it, understanding what is the risks, understanding like just basic stuff like, hey, this is gonna be really challenging because we need a particular kind of data to get this, and we don’t have that data in order for AI to make this kind of inference.

So it’s just understanding AI as a material, I think is one half of it.

The other half is what my colleague Nick Martelaro calls the “AI augmented designer,” where it is the use of AI tools in the design process.

So these are things like being able to do certain kinds of prompts. These are things like being able to create an agent to do certain pieces of the design production work that you may want to do. This is being able to set up a technical environment that you work in, and being able to talk to engineers about how to work with the tech stack that’s there and in order to push stuff to production.

So all those kinds of things, and being able to kind of work fluidly between a tool like Figma that gives you an overview that lets you see the whole system. That’s where your design system is. And then a tool like Claude Code or Figma Make or something like that in order to actually prototype and bring this thing to life in a way that would’ve been impossible five years ago.

So those are the two big skills. One is an understanding of the material, the way that AI works, the probabilistic nature of it, the risks of it. And then the other is how do I work with that as part of my design process?

So having those two things, I think, is what is really valuable to design organizations.

And then of course having the other skills we’ve been talking about, the foundational things, judgment, being able to understand when something is good, being able to figure out what is quality. The combination of those things, I think is a really valuable asset to have on your design team.

Jesse: Well, and another thing that came to mind as you were speaking with all of this is the ability to be, well, adaptive in how you approach things. Not to be too wedded to a particular tool set, not to be too wedded to a particular process, but to be experimental in your approach.

We were talking about the 25th anniversary of Adaptive Path and the experimental nature of how we approached things there, and I wonder if there’s something similar that applies here.

Dan: Yeah, absolutely. I mean the tools change every couple weeks. You know, I mean, it’s pretty funny.

I keep telling the story, but I’ll tell it again, which is, I was teaching my class like, oh look, you can take stuff from Claude Code and put it into Figma. and they’re like, oh, that’s interesting.

And then of course the next question was, well, can you take stuff from Figma and put it back into Claude Code? And I was like, absolutely not. You cannot do that. And then a week later I was like, remember when I told you you couldn’t do that? Guess what? Now you can.

And so there is definitely this idea of, yeah, being willing to be like, oh, now I can do that. Now my process can shift a little to accommodate as the tools get better and better and more sophisticated.

But always knowing that like with any new technology or any new tool, it’s a bumpy ride. it’s not gonna be fully baked all the time.

I mean, the stuff is still buggy and the stuff changes and if you’re doing releases every couple weeks, as a lot of these places are, there’s just unexpected stuff is gonna happen.

So right there is this idea of experimentation, of playfulness and the ability to like throw stuff out and luckily, as designers, we need to be used to that. We need to be used to having a prototype that we’re like, eh, nope, this doesn’t work. And you know, the kill your darlings mindset really helps when everything is in flux. For sure.

Peter: I have to imagine though, it makes it so difficult to teach when the sands are shifting, right? Because usually teaching, right, is built on a corpus of knowledge that has been honed over decades, and now it’s just… all this change is occurring. And so how do you develop the confidence that what you are communicating is worthwhile, right?

Dan: I mean, it’s humbling. And you have to caveat everything when you are talking about this cutting edge stuff like this. This is true.

Now, it may not be true by the end of the semester. I remember my first semester teaching the design for AI products and services class.

That March, every single day of that March, there was a new AI release or big update. Every single day.

And I was like, how do you keep up with this? I mean, you have to be changing things on the fly. You’re adding in new readings, you’re adding in new exercises, new projects.

Like right now, like our final project for that class is all around AI agents, and that is a space that is really up for grabs and the technology is really not ready for prime time. And so how do you teach students to work with a technology that is kind of janky but is gonna improve?

And some of that is, how do you go back to fundamentals? How do you tie it back to basic things that we know?

And so that’s why, for example, task analysis becomes really important again. ‘Cause it’s like, what are we trying to replace here? Can we make sure that the AI can actually do those things? Or what is the point where the AI should intervene in this system? And what are the places where the AI should absolutely not intervene in this system?

And so having those kind of discussions is really important. Even as the top of the ocean is extremely frothy and tumultuous, down at the bottom is, yeah, is very settled.

And so the more you can kind of tie those two things together, the better off you’re gonna be. But it is not easy.

Peter: Well, and something we haven’t really touched on that I think you’ve written about, is AI not as a means in your workflow, but AI as a material that you are shaping for others to use.

And you’re discussing process. When I think about this, I think back to working with Jeff Veen, speaking of Adaptive Path, right?

Who, even before Adaptive Path, would argue for designers letting go of pixel perfect delivery in favor of a more rules-based mindset, ’cause you never knew what browser people were going to have.

Dan: Right. I remember Jeff’s famous dictum, which I always tell my students, which is, if you’re not embarrassed by it, you waited too long to ship it. And so I always talk about that.

Peter: But over time, whether it’s web browsers or mobile, designers were able to get pixel perfect again and really control it.

But now with this new world, it feels like there’s the possibility for things to emerge. And I mean that literally from an emergence standpoint, kind of the evolutionary theory standpoint of a lot of local things happening and then something comes up and you don’t quite know what it’s gonna be.

And I’m wondering if that’s something that you’re seeing or teaching, or how are you helping your students be okay with not specifying to the nth degree what the final output is, but instead setting up conditions that lean towards desired direction but aren’t, you know, putting real firm stakes in the ground of this is what the experience has to be.

AI and the Double Diamond

Dan: Right. So there’s a couple, couple ways to think about how AI is being additive to the classic double diamond process.

Which is that, yeah, at the beginning when you’re doing strategy, you have to think of things like data. You have to think of things like risk, you have to think of things like, if you know upfront that this is an AI project, is this appropriate for AI?

So things like that in the like design strategy portion, thinking about those kinds of things.

But then once you get to that second diamond where you’re talking about design execution, making the thing, right? There’s a lot of kind of new pieces that are starting to fall into place there.

And by new, for some of them, I mean old, in that it is things like designing for intent, trying to figure out what someone’s intent is, which was something that actually came out of conversational UI stuff that those folks have been working on for years. Trying to figure out what is it that people are really asking for here. What are they trying to say and what are they trying to do?

So for example, if I type into a chatbot and say, What’s the best coffee in Pittsburgh? I could be asking for a number of different things. I could be asking for, find me that, I could be asking for, is there a place that will ship it to me? I could be like, is a place near me?

There’s a number of different intents I could have with something that generic. So trying to figure out what intent is and then what is the AI output that best matches that intent?

And what is the best AI input for that too? Is it a chat box? Is it direct manipulation? Is it working behind the scenes to do things like you know, like finding a face in your camera image that it’ll focus on. That’s not something I wanna do manually all the time. That’s a really nice AI trick that can happen in the background.

So figuring out so you’ve got: no AI, you’ve got AI kind of chatbot, and then you have like generated or temporal UI that like, hey, the pieces of whatever you need start to appear. And that’s kind of like what is starting to happen right now.

So is there like a GUI portion of it as well? So there’s that chunk and then there’s the chunk about, like, what is good human- AI collaboration, HAIC, like what are the best practices there about when to ask, when to interrupt, when to just execute, how to explain something, how to say, ” this isn’t working.” Talking about my confidence level in what I’m giving back to you.

All those kinds of things, which are again, still evolving, are things that are affecting the design process as a whole.

And yeah, people don’t talk about those things as much as they should because yeah, they’re focused on the tool part of it.

Like, can you build this thing really fast without thinking about like, well, what are the things that we now need to think about? Because I think people are often thinking like when they’re using these AI powered tools, they’re using them to build traditional software. They are not using them to build AI software, which has a completely different mindset to it.

Peter: Right.

Jesse: So, with all of this stuff unfolding in so many multifaceted directions, what are you most curious to see? What are you most curious about in terms of what happens next?

Dan: I think that there is a couple of interesting points coming up.

We just mentioned one of them, which is, how do I start to get generated widgets or pieces of apps or parts of apps that are displayed as I need them for specific tasks. So this kind of temporal, generative UI pieces.

I think that is something that we’re gonna start seeing more of in the next two to three years, I think maybe even earlier. ‘Cause we’re starting to get all the pieces of those put together right now. So I think that is, I think that’s something that’s coming pretty fast.

I think the thing that’s gonna happen in the next year is that we are gonna see a real merger between Code and Figma, where you’ve, got Figma for that big overview. The ability to put together flows and things that are really hard to do in code but are easier to do visually and with direct manipulation, we are going to start seeing the merger of those where I’m gonna be able to put together a flow in Figma or Figjam and then say, okay, code this flow up so I can test that code.

Okay, great. And then it’ll be like, okay, well apply the design system to this. Okay, great. Well I’m gonna move this button over on this thing because it looks misaligned. Okay, well then that automatically translates into the code. So I think we’re gonna start seeing that kind of code and overview slash flow really start to merge. And I think that’s a really exciting point.

And then I think that the agents, the ability to take action on things and do things on your behalf is going to get better. And I think that will be pretty interesting where you’ll be able to, you know, have it do things for you on your behalf.

Now we’re seeing with things like OpenClaw, we’re seeing some crazy, wild stuff and I think it’s absolutely bonkers that people are installing this on their machines and having it go do stuff for them. It sounds like a nightmare to me.

But I mean, one of the things I think is super interesting about OpenClaw or whatever we’re calling it now, Moltbot… But that it is establishing its own goals and working more proactively in a different kind of back and forth than we see right now.

So right now, in order for me to get AI to do most things, I have to prompt it. I have to say, go do this thing for me. What if it starts to be like, well, I did this thing for you, or, Hey, I’ve noticed this thing happening and I’d like to do this thing for you.

So that it is more of a back and forth dialogue rather than I am always the actor, the AI is a passive servant.

And I think that’s really interesting from a interaction design perspective, from a HCI perspective, from a lot of different, from a security perspective, from a risk perspective, I think that there’s a lot of like weirdness there as we’re seeing with OpenClaw.

But I think it’s a really neat future to design for, and I’m looking forward to starting to think about that.

Jesse: Dan Saffer, thank you so much for being with us.

Dan: Yeah.

Jesse: Anything you wanna plug while you’re here?

Dan: It’s been a pleasure to find our way. I will plug my own podcast, which is AI in Design, which is a little weekly podcast we do, where we recap the week’s news and talk about it with my colleague Nik Martelaro, who is more on the tech side.

So he’s able to explain things to me that I’m like, what is this MCP thing? And he’s like, okay, here’s what it is. So really great for folks in perhaps in your audience who are design leaders and stuff like that, to start to wrap their head around what’s going on at kind of that cutting edge, which is… We try to not be like, oh, look at this news thing that’s happening, or look at this big deal. We’re very focused on the intersection of AI and design. So that’s kind of our niche.

I also have a couple workshops coming up in the Rosenfeld- verse, so that’s happening. Gonna be doing some of those. I think around the Designing for AI conference that’s happening, but also I think before that, like in April or May. And then there’s another one, I think in October that are all happening. And, yeah, that’s my plugs for now.

I am working on a designing for AI products and services book with two of my colleagues, John Zimmerman, Jody Forlizzi. It is, as you may imagine, a very difficult book to write.

Given the circumstances that we’re in right now, how do you start to make this into a thing that doesn’t get outta date the moment that you send it to the printer?

We are really trying to find those fundamental things that are unlikely to change, and that is very hard these days.

Jesse: Well, thank you so much for joining us.

Dan: Yeah. Thanks for having me. It’s, been fun to hang out and catch up.

Jesse: Thanks, Dan.

For more Finding Our Way, visit findingourway.design for past episodes and transcripts, or follow the show on LinkedIn. Visit petermerholz.com to find Peter’s newsletter, The Merholz Agenda, as well as Design Org Dimensions featuring his latest thinking and the actual tools he uses with clients.

If you’re looking for help with AI transformation or you just need a private advisor to help you solve your hardest leadership problems, visit my website at jessejamesgarrett.com to book your free one hour consultation.

If you’ve found value in something you’ve heard today, we hope you’ll pass this episode along to someone else who can use it. Thanks for everything you do for others, and thanks so much for listening.