Why StackOverflow usage is down 50% — with David Hsu of Retool

We are announcing the second edition of our Latent Space demo day event in SF on 2/23: Final Frontiers, a startup and research competition in “The Autonomous Workforce”, ​”Beyond Transformers & GPUs”, and “​Embodied AI”.

RSVP here! The first one was aimed for 15-20 people and ended up blowing up to >200 and covered in the Information - let’s see what a year of growth (and competition) does to the local events space in 2024.

You can find all Latent Space events here, and of course get in touch with us to host your own AI Engineer meetups like AI Engineering Singapore.

In our December 2023 recap we covered the Four Wars of the AI stack. But how do we know when it’s time to crown a winner? As we kick off 2024, we wanted to do a recap of the State of AI in 2023 to set a baseline of adoption for different products. Retool had a great report at the end of last year which covered a lot of it.

David Hsu, CEO and co-founder of Retool, joined us to go over it together. We also talked about the history of Retool, why they were too embarrassed to present at YC demo day, and how they got to $1M ARR with 3 employees. If you’re a founder, there are a lot of nuggets of advice in here!

Retool AI

In our modeling of the “Software 3.0 Stack”, we have generally left a pretty wide open gap as to the “user interface” equivalent of the AI stack:

Retool AI launched 4 months ago with some nifty features for SQL generation, and its own hosted vector storage service (using pgvector). However, as he explains on the pod, the more interesting potential of Retool is in helping developers build AI infused applications quickly, in combination with its Workflows feature.

This moves Retool down the stack from just the UI for internal tooling to the business logic “piping” as well. There are a bunch of dedicated tools in this space like Respell, BuildShip, Flowise, and Ironclad Rivet.

"We think that practically every internal app is going to be AI infused over the next three years." - David on the pod

RIP StackOverflow?

In July 2023 we talked about the impact of ChatGPT and Copilot:

This was then disputed by StackOverflow, who pointed out (very fairly so) that there were privacy-related changes in their analytics instrumentation in 2022. StackOverflow no longer reports traffic, but based on StackOverflow’s continuing transparency we can see that organic declines have continued throughout 2023.

Retool’s report comes over a year after those changes and has some self reported samples from users:

* 57.6% of people said they have used StackOverflow less; almost all of them replaced it with ChatGPT and Copilot.

* 10.2% said they no longer use StackOverflow.

We also saw a lot more tools being released in the dev tools space such as (one of our oldest pod friends) Codeium (which just raised a $65M Series B), SourceGraph (and their newly released Cody), Codium AI (just released AlphaCodium which was picked up by Karpathy), Phind (which beat GPT-4 with OSS models), and Cursor, one of the most beloved products in the dev community at the moment. Intelligence is getting closer and closer to the IDE, and the trend doesn’t seem to be reverting.

We already said that “You are not too old (to pivot into AI)“, and the advice still stands. When asked to rate “Preference for hiring engineers effective at using ChatGPT/Copilot for coding” on a scale of 1 to 10, where 10 is “Much more likely”, ~40% of companies voted 8-10. Having an AI Engineer skillset is extremely important. 45% of companies between 1,000-4,999 employees said that they increased the difficulty of technical interviews to compensate for these new tools, so the gap between users and non-users will keep widening.

Crossing the AI in Production Chasm

Geoffrey Moore’s “Crossing the Chasm” is one of the most quoted business frameworks. Every market has an initial group of Innovators and Early Adopters, who are willing to suffer through the rough edges of products initially, and eventually crosses into the Early Majority, which expects a full product.

In the AI world, ChatGPT and Midjourney / DALL-E have crossed the chasm in the consumer space. Copilot is probably the only tool that did it in the enterprise, having crossed 1M paid users. ~$50B were invested in AI in 2023, and we still only have <5 breakout products; expect this number to rise in 2024. According to the survey, only 25% of companies had real production usage, but 77.1% said their company is making efforts to adopt more. Closing that gap could triple AI adoption in one year.

The report also broke down adoption by use case. 66% of companies use it internally, while only 43% do so in customer-facing use cases. Internal usage of AI is much more varied than customer-facing one as well:

One point that David made in the podcast is that this number isn’t a knock on AI as a tool, but rather about the demographics of businesses outside of our Silicon Valley bubble:

We all work in Silicon Valley, right? We all work at businesses, basically, that sell software as a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. […] Most of the [work of] software engineers in the world actually goes towards these internal facing applications.

Beyond code models, it’s clear that the big winners of the first wave of AI adoption are vector stores and RAG. Knowledge base Q&A, customer chatbots, recommendation systems, etc are all based on them. Retool even rolled out their own with Retool Vectors. Expect the battlefield to get even hotter in these areas, with Mongo and Chroma leading the charge on a NPS/popularity basis.

It’s also clear that OpenAI won the first campaign in the AI models war, by far. Hopefully Mistral and LLaMA3 will shake up this chart when we look back at it in 2025:

TLDR: We’re really early. If you want to build in AI, there’s a ton of work to be done, and a lot of problems to be solved. You can find the full report here to dive through all the numbers.

Video podcast

Watch along on our snazzy YouTube!

Show Notes

Companies and Projects:

* Retool

* State of AI Report

* Retool AI

* Retool Workflows

* Raising less money at lower valuations

* Paul Graham's "playing house" essay

* Gödel, Escher, Bach (GEB)

Timestamps

* [00:00:00] Introduction

* [00:02:43] Retool's founding story and decision not to present at YC demo day initially

* [00:09:08] Philosophy on fundraising - raising less money at lower valuations

* [00:12:53] Overview of what Retool is

* [00:15:41] Origin story of Retool AI product

* [00:19:59] Decision to use open source vector database PG Vector

* [00:21:29] Most underrated AI use cases

* [00:25:56] Retool's AI UX and workflows

* [00:30:38] Zapier vs Retool

* [00:32:54] Updates from Retool's 2023 State of AI survey

* [00:35:21] Who is adopting AI first?

* [00:37:40] Evolving engineering hiring practices in the age of Copilot/ChatGPT

* [00:40:02] Retool's views on internal vs external AI adoption

* [00:41:50] OSS models vs OpenAI in production

* [00:44:46] Additional survey questions to ask in 2024

* [00:47:04] Balancing enterprise sales vs bottom-up adoption

* [00:51:54] Philosophical thoughts on AGI and intentionality

Transcript

Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.

Swyx [00:00:16]: And today we are in the studio with David Hsu from Retool. Welcome.

David [00:00:20]: Thanks. Excited to be here.

Swyx [00:00:23]: We like to give a little bit of intro from what little we can get about you and then have you talk about something personal. You got your degree in philosophy and CS from Oxford. I wasn't aware that they did double degrees. Is that what you got?

David [00:00:35]: It's actually a single degree, which is really cool. So basically you study content, you study philosophy, and you study intersection. Intersection is basically AI, actually, and sort of computers think, or computers be smart. What does it mean for a computer to be smart? As well as logic. It's also another intersection, which is really fun too.

Swyx [00:00:51]: In Stanford, it might be symbolic systems or whatever. It's always hard to classify these things when we don't really have a word for it. Now I guess everything's just called AI. Five years ago, you launched Retool. You were in YC at winter 17 and just been a straight line up from there, right?

David [00:01:09]: I wish.

Swyx [00:01:10]: What's something on your LinkedIn that people should know about you? Maybe on their personal hobby or, you know, let's just say something you're very passionate about.

David [00:01:17]: Yeah, sure. I read quite a bit. I probably read like two books a week around about. So it's a lot of fun. I love biking. It's also quite a bit of fun. So yeah.

Swyx [00:01:25]: Do you use Retool to read?

David [00:01:27]: No, I don't use Retool to read. No, that'd be funny.

Swyx [00:01:30]: What do you read? How do you choose what you read? Any recommendations?

David [00:01:35]: I'm mostly reading fiction nowadays. So fiction is a lot of fun. I think it helps me be more empathetic, if you will. I think it's a lot of fun. I actually just want to see what it's like to be in someone else's shoes. That's what I really good about philosophy as well. I find philosophy just so interesting, especially logic. We can talk more about that for probably hours if you want.

Swyx [00:01:50]: So yeah, I have a casual interest in epistemology. And I think that any time you, you know, you're trying to figure out a way to solve a problem, you're going to have to figure out a way to solve it.

David [00:02:05]: Yeah, totally. What does it mean to know?

Alessio [00:02:13]: That's its own podcast. We should do a special edition about it. That's fun. Let's maybe jump through a couple of things on Retool that I found out while researching your background. So you did YC, but you didn't present a demo day initially because you were too embarrassed of what you had built. Can you maybe give any learnings to like founders or people who are built? I've seen a lot of people kind of give up early on because they were like, oh, this isn't really what I thought it was going to be to be a founder. They told me I would go to YC and then present and then raise a bunch of money and then everything was going to be easy. So how did that influence also how you build Retool today in terms of picking ideas and deciding when to give up on it?

David [00:03:30]: Yeah. Let's see. So this is around about 2017 or so. So we were supposed to present at the March demo day, but then we basically felt like we had nothing really going on. We had no traction, we had no customers. And so we're like, okay, well, why don't we take six months to go find all that before presenting? Part of that, to be honest, was I think there's a lot of noise around demo day, around startups in general, especially because there's so many startups nowadays. And I guess for me, I'd always wanted to sort of under-promise and over-deliver, if you will. And then demo day, I mean, maybe you two have seen a lot of videos. It's a lot of, honestly, over-promising and under-delivering because every startup says, oh, I'm going to be the next Google or something. And then you peer under it and you're like, wow, nothing's going on here, basically. So I really didn't want that. And so we chose actually not to present on demo day, mostly because we felt like we didn't have anything substantial underneath. Although actually a few other founders in our batch probably would have chosen to present in that situation, but we were just kind of embarrassed about it. And so we basically took six months to just say, okay, well, how do we get customers? And we're not presenting until we have a product that we're proud of and customers that we're proud of. And fortunately, it worked out. Six months later, we did have that. So I don't know if there's much to learn from the situation besides I think social validation was something that I personally had never really been that interested in. And so it was definitely hard because it's hard to sort of, it's almost like you go to college and all your friends are graduating, but you failed or something, you failed the final and you have to like redo it here. It's like, well, it kind of sucks that all your friends are up there and on the podium presenting and they are raising a ton of money and you're kind of being left behind. But in our case, we felt like it was a choice. We could have presented if we really wanted to, but we would not have been proud of the outcome or proud of what we were presenting. And for us, it was more important to be true to ourselves, if you will, and show something that we're actually proud of rather than just raise some money and then shut the company down in two years.

Alessio [00:04:45]: Any Sam Altman stories from the YC days? Could you tell in 2017 that Sam was going to become, like, run the biggest AI company in the world?

David [00:04:49]: Wow. No one's asked me that before. Let me think. Sam was, I think he was, I want to, I forgot, I think maybe president of YC in our batch. We actually weren't in his group actually at the very beginning. And then we got moved to a different group. I think Sam was clearly very ambitious when we first met him. I think he was very helpful and sort of wanted to help founders. But besides that, I mean, I think we were so overwhelmed by the fact that we had to go build a startup that we were not, you know, honestly paying too much attention to everyone else's partner taking notes on them.

Alessio [00:05:20]: That makes sense. Well, and then just to wrap some of the Retool history nuggets, you raised a series A when you were at 1 million in revenue with only three or four people. How did you make that happen? Any learnings on keeping teams small? I think there's a lot of overhiring we've seen over the last few years. I think a lot of AI startups now are raising very large rounds and maybe don't know what to do with the capital.

David [00:05:42]: So this is kind of similar, actually, from sort of why we choose not to demo day. And the reason was, it feels like a lot of people are really playing startup. I think PG has an essay about this, which is like, you're almost like playing house or something like that. Like, it's like, Oh, well, I hear that in a startup, you're supposed to raise money and then hire people. And so therefore you go and do that. And you're supposed to, you know, do a lot of PR, because that's what, you know, startup founders do. And so you could do a lot of PR and stuff like that. And for us, we always thought that the point of starting a startup is basically you have to create value for customers. If you're not creating value for customers, like everything else is going to, nothing's going to work. Basically, you can't, you know, continuously raise money or hire people if you don't have customers. And so for us, we were always very focused on that. And so that's initially where we started. I think it's, again, maybe goes to like the sort of presenting something truthful about yourself or staying true to yourself is something to that effect, which is we didn't want to pretend like we had a thriving business. And so the only way to not pretend was actually to build a thriving business. And so we basically just, you know, put our heads down and, you know, grinded away for probably a year, year and a half or so, just writing code, talking to customers. And I think that at that point we had raised something like maybe a million dollars, maybe a million and a half, something out of YC. So I mean, to us, to people, you know, that was a huge amount of money. I was like, wow, like, how are we ever going to spend a million and a half? The runway was like, you know, five, six years at that point, right? Because we're paying ourselves 30, 40K a year. And so then the question was not like, oh, we're going to run on the runways. The question was like, we better find traction because if we don't find traction, we're going to, you know, just give up psychologically. Because if you run an idea for four years and nothing happens, you're probably psychologically going to give up. And I think that's actually true in most startups, actually. It's like most startups die in the early stages, not because I run out of money, but really because you run out of motivation. And for us, had we hired people, I think it would have actually been harder for us because we want to run out of motivation faster. Because when you're pre-product market fit, actually, trying to lead the team of like, you know, 10 people, for example, to Marshall's product market fit, I think it's actually pretty hard. Like it's, you know, every day people are asking you, so why are we doing this? And you're like, I don't know, man, like, hey, trust this. That's actually a very tiring environment to be in. Whereas this is just like, you know, the founders figuring out product market fit, I think it's actually a much sort of safer path, if you will. You're also schooling less with employees, like when you hire employees, you have an idea you're trying to work with your customers. That's actually, I think, a lot more stable of a place for employees to join as well.

Swyx [00:08:00]: Yeah. I find that typically the sort of founder employee relationship is, employee expects the founder to just tell them what to do, and you don't really get critical pushback from the employee, even if they're a body and even if they like you as an early engineer. It's very much like the role play of like, once you have that founder hat on, you think differently, you act differently, and you're more scrappy, I guess. In trying to figure out what that product is. Yeah, I really resonate with this, because I'm going through this right now.

David [00:08:26]: Awesome. One thing we did actually early on that I think has paid a lot of dividends, especially your rituals a lot larger now is we hired a lot of former founders. So I want to say like, when we were 20, 30, 40 people, we were probably like half former founders at each one of those stages. And that was actually pretty cool, because I think you infuse sort of a, you know, get things done kind of culture, a outcome oriented culture of like a very little politics, because you know, no one came from larger companies, everyone was just like, this is my own startup, let me go figure out how to achieve the best outcome for the customer. And so I think from a cultural perspective, even today, a lot of rituals culture is sort of very self startery. I think it's actually because of sort of these like, you know, early founders that we hired, which was really, really, you know, we're really lucky to have had them. Yeah.

Swyx [00:09:08]: And then closing off on just a little bit of the fundraising stuff, something notable that you did was when in 2021, when it was sort of peak Zerp, and everyone was raising hundreds and hundreds of millions of dollars, you intentionally raised less money at lower valuations as your title. And I think it's a testament to your just overall general philosophy and building retool that you're just very efficient and you do things from first principles. Any updates on like, would you still endorse that? You know, would you recommend that to everyone else? What are your feelings sort of two years on from that?

David [00:09:38]: Yeah. I think exactly you said is correct, where we raise less money at a lower valuation. And I think the funny thing about this is that when we first announced that, even, you know, internally and both externally, I think people were really surprised, actually, because I think Silicon Valley has been conditioned to think, well, raising a giant sum of money at a giant valuation is a really good thing. So like, you know, you should maximize both the numbers, basically. But actually maximizing both the numbers is actually really bad, actually, for the people that matter the most, you know, i.e. your employees or your team. And the reason for that is raising more money means more dilution. So if you look at, you know, a company like, let's say Uber, for example, if you join Uber at like, I don't know, like a $10 billion valuation, or let's say join for a huge route, which I think happened at a few billion dollars in valuation, you actually got diluted a ton when Uber fund raises. So if Uber raises, if Uber dilutes themselves by 10%, for example, let's say it raised $5.25 billion, for example, I think employees' stake goes down by 10% in terms of ownership. Same with, you know, previous investors, same with the founders, etc. And so, if you look at actually a lot of founders in sort of, you know, the operations statistics space, or, you know, those that fundraise, like, you know, 2013, 2017, a lot of the founders by IPO only have a few percentage points, actually, for a company. And if founders only have a few percentage points, you can imagine how, you know, how little employees have. And so, that I think is actually still really, you know, bad thing for employees overall. Secondly, sort of higher valuation, given the same company quality is always worse. So basically, what that means is, if you are fundraising as a company, you could commit a certain valuation in the market, you know, let's say it's, you know, x. For example, maybe if you're lucky, and you can raise two times x, for example. But if you choose two times x, your company itself is not fundamentally changed. It's just that, you know, for some reason, investors want to pay more for it. You know, maybe today you're an AI company, for example. And so, investors are really excited about AI and want to pay more for it. However, that might not be true in a year or two years time, actually. And if that's not true in two years time, then you're in big trouble, actually. And so, now, I think you see a lot of companies that are raising really high valuations about 2021. And now, they're like, man, we're at like 100x, or, you know, we raised 300x multiple, for example. And if we're at 300x, then, you know, maybe now we're at like 200x, man, we just can't raise money ever again. Like, you know, we have to grow like 50x to go raise money, reasonable valuation, I would say. And so, I think that is really challenging and really demotivating for the team. And so, I think a lower valuation actually is much better. And so, for us, in retrospect, you know, to answer your question, two years later, we did not predict, you know, the crash, if you will. But given that, I think we've done extremely well, mostly because our valuation is not sky high. If our valuation were sky high, I think we'd have a lot more problems. We'd probably have recruiting problems, for example, and probably have a lot of internal morale problems, etc. A lot of people would be like, you know, why is the valuation this way? We might have cash flow problems because we might have to go raise money again, you know, etc. But we can't because the valuation is too high. So, I would urge, I think, founders today to, quote unquote, like, leave money on the table. Like, there are some things that are not really worth optimizing. I think you should optimize for the quality of the company that you build, not like the valuation, you raise that or the amount you raise, etc. So,

Swyx [00:12:34]: Highlight 2020, but it looks like, you know, you made the right call there anyway. Maybe we should also, for people who are not clued into Retool, do a quick, like, what is Retool? You know, I see you as the kings or the inventors of the low-code internal tooling category. Would you agree with that statement? You know, how do you usually explain Retool?

David [00:12:53]: I generally say it's like Legos for code. We actually hate the low-code moniker. In fact, we have docs saying we will never use it internally, or even to customers. And the reason for that is, I think, low-code sounds very not developer-y. And developers, they hear the phrase low-code, they're like, oh, that's not for me. I love writing code. Like, why would I ever want to write less code? And so, for us, Retool is actually built for developers, like, 95% of our customers actually are developers, actually. And so, that is a little bit surprising to people. I'll generally explain it as, and this is kind of a funny joke too, I think part of the reason why Retool has been successful is that developers hate building internal tools. And you can probably see why. I mean, if you're a developer, you've probably built internal tools yourself, like, it's not a super exciting thing to do, you know, it's like piecing together a CRUD UI, you've probably, you know, pieced together many CRUD UIs in your life before, and there's a lot of grunt work involved. You know, it's like, hey, state management, it's like, you know, data validation, it's like displaying error messages, it's like the bouncing buttons, like, all these things are not really exciting. But you have to do it, because it's so important for your business to have high quality internal software. And so what Retool does is basically allows you to sort of, really fast, whether it's a front end, whether it's a back end or whatever else. So yeah, that's what Retool is.

Swyx [00:14:02]: Yeah, actually, if you started hiring, and so I do a lot of developer relations and community building work, and you hired Krithika, who is now, who's also an AI, to start out your sort of DevRel function. And I was like, what is Retool doing courting developers? And then she told me about this, you know, developer traction. And I think that is the first thing that people should know is, which is that the burden and the weight of internal tooling often falls to developers, or it's an Excel sheet somewhere or whatever. But yeah, you guys have basically created this market, you know, in my mind, I don't know if there was someone clearly before you in this, but you know, you've clearly taken over and dominated. Every month, there's a new YC startup launching with that it's like, you know, we're the open source Retool, we're like the lower code Retool, whatever. And it's pretty, I guess it's endearing, you know, we'll talk about Airplane later on. But yeah, I think I've actually used Retool, you know, in my previous startups for this exact purpose. Like, we needed a UI for AWS RDS that they can, you know, like the rest of our non less technical people, like our sales operations, people could could interact with and yeah, Retool is perfect for that.

David [00:15:04]: Yeah, it's a good example of like, that's an application that an engineer probably does not want to build, like building an app on top of Salesforce or something that is not exciting. And so it sucks. It's very limited. It's like not a fun experience at all. But piecing together a Retool is quite a bit easier. So yeah, let me know if you have any feedback, but also, yeah, no, of course, like more recently,

Swyx [00:15:23]: I think about three, four months ago, you launched Retool AI, obviously, AI has been sort of in the air. I'd love for you to tell the journey of AI products ideation within Retool. Given that you have a degree in this thing, I'm sure you're not new to this, but like, when would you consider sort of this the start of the AI product thinking in Retool?

David [00:15:41]: So we actually had a joke internally at Retool. We are part of roadmap for every year, it was like 2019 or something. We had this joke, which was like, what are we going to build this year? We're going to build AI programming, is what we always said as a joke. And so, but it was funny, because we were like, that's never gonna happen. But like, let's add it because it's like a buzzword thing that enterprises love. So let's look at it. And so it was almost like a funny thing, basically. But it turns out, you know, we're actually building that now. So this is pretty cool. So I would say maybe AI thinking on Retool probably first started maybe like, I don't know, a year and a half ago, something like that. And when we first started thinking about it, sort of in a philosophical way, if you will, it's like, well, what is the purpose of AI? And how can it help, you know, what Retool does? And there were two main prongs, if you will, value. One was helping people build apps faster. And so you've probably seen Copilot, you've seen sort of so many other coding assistants, P0 to them, you know, stuff like that. So that's interesting, because, you know, engineers, as we talked about, do some grunt work. And grunt work, you know, maybe could be automated by AI was sort of the idea. And it's interesting. So we actually, I would say, kind of proved or disproved the hypothesis a little bit. If you talk to most engineers today, like a lot of engineers do use Copilot. But if you ask them, like, how much time does Copilot save you? It's not like coding is 10x faster than before, you know, coding is maybe like 10% faster, maybe 20% faster, or something like that, basically. And so it's not like a huge step change, actually. And the reason for that, as we think, is because the sort of fundamental frameworks and languages have not changed. And so if you're building, let's say, you know, like the sales ops tool we're talking about before, for example, let's say you've got AI to generate you a first version of that, for example, the problem is that it probably generated it for you in like JavaScript, because you're writing for the web browser, for example, right. And then for you to actually go proofread that JavaScript, for you to go read the JavaScript to make sure it's working, you know, to fix the subtle bugs that AI might have caused, hallucinations, stuff like that, actually takes a long time and a lot of work. And so for us, the problem is actually not like the process of coding itself, it is more sort of the language or the framework, I think it's like way too low level. It's kind of like anything like punched cards, like, let's say, back in the day, who designed punched cards, and AI could help you generate punched cards, okay, you know, I guess that helps me punch the cards a little bit faster, because I have a machine punching them for me. But like, when there's a bug, I still have to go read all the punched cards and figure out what's wrong, right? It's like, it's a lot of work, actually. And so, for us, that was the sort of initial idea was, can we help engineers code faster? You know, I think it's somewhat helpful, to be clear, like, again, I think it's 10 or 20%. So we have things like, you know, you can generate school careers by AI, you can generate UIs by AI, and stuff like that. So that's cool, to be clear. But it's not, I think, the step change, that I think is, you know, the, we're investing somewhat in that. But the bulk of investment, actually, is a number two, which is helping developers build AI enabled applications faster. And the reason why we think this is so exciting is we think that practically every app, every internal app, especially, is going to be AI infused over the next, like, three years. And so every tool you might imagine, so like the tool you were mentioning, like a sales operations tool, for example, probably, you know, if you were to build today, one of the corporate subform of AI. And so, you know, we see today, like, for us, like, a lot of people build, you know, I'll say sales manager tools, or retool. An example is there's a fortune, like a company is building like, sales forecasting tools. So they basically have salespeople enter their forecast, you know, for the quarter, the beginning of the quarter, like, hey, I have these deals. And these deals are going to close, these deals are not going to close, you know, I think I'm upsiding these, downsiding these, stuff like that, basically. So you can imagine it's pulling in deals from your Salesforce database. And so it pulls in the deals that actually use AI to compute like, okay, well, you know, given previous deal dynamics, like, these are the deals that are more likely to close this month versus next month was this quarter, next quarter, etc. And so it could actually, you know, pre write you a draft of, you know, your report, basically. And so that's an example where I think all apps, whether it's, you know, a sales app, you know, until it looks like fraud app, a, you know, fintech app, you know, whatever it is, basically, especially internal apps, I think, like you said, Alessio, in order to make you more productive, it's going to incorporate some form of AI. So the other question is, can we help them incorporate AI faster? So that's why we launched like a vector database, for example, built directly into retool. That's why we know launches all these AI actions, you don't have to go figure out what the best model is and do testing and stuff like that, which gives you out of the box. So for us, I think that is really the really exciting futures. Can we make every app and also retools use AI a little bit and make people more productive?

Alessio [00:19:59]: So for Wang, who's the co founder and chief architect of amplitude, he mentioned that you just use Postgres vector. When you were building retool vectors, how do you think about, yeah, leveraging a startup to do it, putting vectors into one of the existing data stores that you already had? I think like, you're really a quite large customer scale. So like, you're maybe not trying to get too cute with it. Any learnings and tips from that?

David [00:20:23]: Yeah, I think a general philosophical thing I think we believe is, um, we think the open source movement in AI, especially when it comes to all the supporting infrastructure is going to win. And the reason for that is we look at like developer tools in general, especially for such a fast moving space. In the end, like, there are really smart people in the world that have really good ideas, and are going to go build companies and they're going to go build projects basically around these ideas. And so for us, we have always wanted to partner with maybe more open source providers or projects, you could say, like PG factory, for example. And the reason for that is it's easy for us to see what's going on under the hood. A lot of this stuff is moving very fast. Oftentimes, there are bugs, actually. And so we can go look and fix bugs ourselves and contribute back to them, for example. But we really think open source is going to win in this space. It's hard to say about models. I don't know about models necessarily, because it's going to be pretty complicated there. But when it comes to tooling, for sure, I think there's just like so much, there's an explosion of creativity, if you will. And I think betting on any one commercial company is pretty risky. But betting on the open source sort of community and the open source contributors, I think is a pretty good bet. So that's why we decided to get at consumer games. Awesome.

Alessio [00:21:29]: So we're going to jump into the survey next, but we're going to put a bunch of links in the show notes about Result AI and whatnot. Is there any most underrated feature, like something that customers maybe love that you didn't expect them to really care about? I know you have a like text to SQL, you have UI generation, there's like so many things in there. Yeah. What surprised you?

David [00:21:49]: Yeah. So what's really cool, and this is my sense of the AI space overall, you know, if you're on YouTube as well, is that, especially in Silicon Valley, where a lot of the innovation is happening, I think there's actually not that many AI use cases, to be honest. And AI to me, even as of January 19th of 2024, still feels like in search of truly good use cases. And what's really interesting, though, about Retool, and I think we're in a really fortunate position, is that we have this large base of sort of customers, and a lot of these customers are actually much more legacy, if you will, customers. And a lot of them actually have a lot of use cases for AI. And so to us, I think we're almost in like a really perfect or unique spot, we're able to adopt some of these technologies and provide them to some of these like older players. So one example that actually really shocked and surprised me about AI was, so we have this one clothing manufacturer, I think it's either the first or second largest clothing manufacturer in the world, who's using Retool. They're a ginormous company, very multinational, stores on pretty every mall in the world. And so they have one problem, which is, they need to design styles every year, for the next year, basically, for every season. So like, hey, just like summer 2024, for example, and we're going to design. And so what they used to do before is they were hiring designers, and designers would go to study data, they'd be like, okay, well, it looks like, you know, big floral patterns are really hot. And like, you know, California, for example, in 2023, and like, do I think it's going to be hot in 2024? Well, let me think about it. I don't know. Maybe. And if so, if I believe it's going to be hot, let me go design some floral patterns, actually. And what they ended up doing in Retool, actually, is they actually automated a lot of this process away in Retool. So they actually now built a Retool app that allows actually a non-designer, so like an analyst, if you will, to analyze like, you know, who are the hottest selling patterns, you know, particular geos, like this was really hot in Brazil, this is really hot in China, it's really hot, you know, somewhere else, basically. And then they actually feed it into an AI. And the AI, you know, actually generates with Dolly and other image generation APIs, actually generates patterns for them. And they print the patterns, which is really cool. And so that's an example of like, honestly, a use case I would have never thought about, like thinking about like, you know, how clothing manufacturers create their next line of clothing, you know, for the next season, like, I don't know, I never thought about it, to be honest, nor did I ever think, you know, how it would actually happen. And the fact that they're able to leverage AI and actually, you know, leverage multiple things in Retool to make that happen, is really, really, really cool. So that's an example where I think if you go deeper into sort of, if you go outside the Silicon Valley, there are actually a lot of use cases for AI. But a lot is not obvious, like you have to get into the businesses themselves. And so I think we're, we personally are in a really fortunate place. But if you know, you're working in the space and want to find some use cases, please come talk to us like, you know, we're really excited about marrying sort of technology with use cases, which I think is actually really hard to do right now.

Swyx [00:24:38]: Yeah, you know, I have a bunch of like, sort of standing presentations around like, how this industry is developing. And like, I think the foundation model layer is understood. The chain vector DB rag layer is understood, I always have a big question mark and actually have you and Vercel V0 in that box, which is like sort of the UI layer for AI. And like, you know, you are perfectly placed to expose those functionalities to end users, you personally don't really know what they're going to use it for. And sometimes they'll surprise you with their creativity. One segment of this, I do see some startups springing up to do this is related to the things that to something that you've you also build, but it's not strictly AI related, which is retool workflows, which is the sort of canvassy boxes and arrows point and click do this then do that type of thing like which which every what are we calling low code? Every internal tooling company eventually builds, you know, I worked at a sort of workflow orchestration company before, and we were also discussing internally how to make that happen. But you are you're obviously very well positioned to it to that. Yeah, basically, like, do you think that there is an overlap between retool workflows and AI? I think that, you know, there's there's a lot of interest in sort of chaining AI steps

David [00:25:55]: together.

Swyx [00:25:56]: I couldn't tell if like that is already enabled within retool workflows, I don't think so. But you could you could sort of hook them together as kind of jankily, like, what's the interest there? You know, is it all of a kind, ultimately, in your mind?

David [00:26:07]: It is 100% on time. And yes, you can actually already saw a lot of people actually are building AI workflows down retool, which is what we're gonna talk about in a second. But a hot take here is actually, I think a lot of the utility in AI today, I would probably argue 60 70% of the utility, like, you know, businesses have found an AI is mostly via chat GPT, and across the board. And the reason for that is, I mean, the chat GPT is sort of a UI, you could say, or interface and user experience is just really quite good, you know, you can sort of converse, you know, with an AI, basically. But that said, there are downsides to it. If you talk to like a giant company, like a J.P. Morgan Chase, you know, for example, they may be reticent to have people copy paste data into chat GPT, for example, even on chat GPT Enterprise, for example. Some problems are that I think chat is good for one off tasks. So if you're like, hey, I want a first version of representation or something like that, you know, and help me write this first version of a doc or something like that, chat is great for that. It's a great, you know, very portable, you know, if you will form factor, so you can do that. However, if you think about it, you think about some economic productivity, more generally, like chat, again, will help you like 10 or 20%. But it's unlikely that you're going to replace an employee with chat, you know, you're not gonna be like, oh, I'm a relationship manager at J.P. Morgan Chase, and I've replaced them with an AI chatbot. It's kind of hard to imagine, right, because, like, the employees are doing a lot of things besides, you know, just, you know, generating, you know, maybe another way of putting it is like, chat is like a reactive interface, like, it's like, when you have an issue, you will go reach out to chat and chatbot solve it. But like, chatbot is not going to solve 100% of your problems, it'll solve like, you know, 25% of your problems, like, pretty quickly, right. And so what we think the next like, big breakthrough in AI is, is actually like automation. It's not just like, oh, I have a problem, let me go to a chatbot and solve it. Because like, again, like, people don't spend 40 hours a week in a chatbot, they spend like two hours a week in a chatbot, for example. And so what we think can be really big, actually, is you're able to automate entire processes via AI. Because then you're really realizing the potential of AI, it's like, not, it's not just like, you know, a human copy pasting data into an AI chatbot, you know, pasting it back out or copying back out. Instead, it's like the whole process now was actually done in an automated fashion without the human. And that, I think, is what's going to really unlock sort of big canonical productivity, or that's what I'm really excited about. And I think part of the problem right now is, I'm sure you all thought a lot about agents is that the agents are actually quite hard. Because like, you know, the AI is wrong, like, you know, 2% of the time, but then you like, you know, a score, if you let's say, you know, raise to the power seven, for example, that's actually wrong, you know, quite often, for example. And so what we've actually done with workflows is we prefer, we've learned, actually, is that we don't want to generate the whole workflow for you by AI. Instead, what we want you to do, actually, is we want you to actually sort of drag and drop the workflow yourself. Maybe you can get a vSphere or something by AI, but it's coded, basically, you should actually be able to modify the steps yourself. But every step can use AI. And so what that means is like, it's not the whole workflow is created by AI, every step is AI automated. And so if you go back to, for example, like the users are talking about, you know, with a clothing manufacturer, that's actually a workflow, actually. So basically, what they say is, hey, every day, we see all the data, you know, from our sales systems into our database. And then we do some data analysis, and, you know, it's just raw SQL, basically, it's nothing too surprising. And then they use AI to generate new ideas. And then the analysts will look at the new ideas and approve or reject them, basically. And that is like a, you know, that's true automation. You know, it's not just like, you know, a designer, copy pasting things as a chat, you can be like, hey, you know, give me a design. It's actually designs are being generated and generated 10,000 designs every day. And then you have to go and approve or reject these designs, which I think is a lot, you know, that's a lot more economically productive than just copy pasting something. So we think sort of the AI workflow space is a really exciting space. And I think that is the next step in sort of delivering a lot of business value by AI. I personally don't think it's, you know, AI chat or AI agents quite yet, so.

Swyx [00:29:50]: That's a pretty reasonable take. It's disconcerting, because like, I know a lot of people trying to build what you already have in workflows. So you have that sort of, you're the incumbent sort of in their minds, I'm sure it doesn't feel that way to you. But like, I'm sure, you know, you're the incumbent in their minds, and they're like, okay, like how do I, you know, compete with retool or, you know, differentiate from retool. As you mentioned, you know, all these connections, it does remind me that you're running up against Zapier, you're running up against maybe Notion in the distant future. And yeah, I think that there'll be a lot of different takes at this space and like whoever is best positioned to serve their customer in the way that they need to shape is going to win. Do you have a philosophy against around like what you won't build, like what do you prefer to partner and not build in-house? Because I feel like you build a lot in-house.

David [00:30:38]: Yes, there's probably two philosophical things. So one is that we're developer first. And I think that's actually one big differentiator between us and Zapier and Notion, and we're so very rare we'll see them actually, and the reason is we're developer first. Because developers, like, if you're like building a sales ops tool, you're probably not considering Notion if you're a developer, you're probably like, I want to build this via React, basically, or use retool. And so are you we build for developers, it's pretty interesting, actually, I think one huge advantage of some of the developers is that developers don't want to be given an end solution. They want to be given the building blocks liquid to themselves to build the end solution. And so for us, like, interesting point that equilibrium we don't get to, it's basically to say, hey, retool is a consulting company, and we basically build apps for everybody, for example. And what's interesting is, we've actually never gotten to that equilibrium. And the reason for that is for some of the developers, developers don't want, you know, like a consultant coming in and building all the apps for them. Developers like, hey, I want to do it myself, just give me the building blocks, give me the best table library, give me, you know, good state management, give me an easy way to query the rest of the API. So I'll do it myself, basically. So that is pretty, so we generally end up basically always building building blocks that are reusable by multiple customers. We have, I think, basically never built anything specific for one customer. So that's one thing that's interesting. The second thing is when it comes to sort of, you know, let's say like, in the AI space, we're going to build and we're not going to build, we basically think about whether it's all core competency or whether there are unique advantages to us building it or not. And so we think about the workflows product, we think workflows actually is a pretty core competency for us. And I think the idea that we can build a developer first workflows automation engine, I mean, I think after we released, you know, workflows, virtual workflows, there have been a sort of few copycats that are, I think, quite, quite far behind, actually, they sort of are missing a lot of more critical features. But like, if you look at the space, it's like, Zapier on one side, and then maybe like, Airflow on the other. And so virtual workflows actually is fairly differentiated. And so we're like, okay, we should go build that. This is the one I was going to build, so I'm just going to build it. Whereas if you look at like vectors, for example, you look at vectors like, wow, there's a pretty thriving space already, if you know vector databases. Does it make sense for us to go build our own? Like, what's the benefit? Like, not much, we should go partner with or go find technology off the shelf. Narcissus is pretty effective. And so for us, I think it's like, how much value does that for customers? Do we have a different take on the space? Do we not? And every product that we've launched, we've had a different take on the space and the products that we don't have a different take, we just adopt what's off the shelf.

Alessio [00:32:54]: Let's jump into the state of AI survey that you ran, and maybe get some live updates. So you surveyed about 1600 people last August, and I were this busy like five years ago. And there were kind of like a lot of interesting nuggets and we'll just run through everything. The first one is more than half the people, 52% said that AI is overrated. Are you seeing sentiment shift in your customers or like the people that you talk to, like as the months go by? Or do you still see a lot of people? Yeah, that are not in Silicon Valley, maybe say, hey, this is maybe not as world changing as you all made it sound to be.

David [00:33:30]: Yes, we're actually on the survey again, actually, in the next few months. So I can let you know when it changes. It seems to me that it has settled down a bit in terms of sort of the maybe like, I don't know, signal to noise, you could say like, it seems like there's a little bit less noise than before. I think people are still trying to look for use cases. I'm saying, but honestly, last year, like United States, again, and I think there are slightly more use cases, but still not substantially more. And I think as far as we can tell, a lot of the surveys, especially some of the comments that we saw, do feel like the companies are investing quite a bit in AI, and they're not sure where it's going to go yet. But they're like, right, it could be big. So I think we should keep on investing. I do think that based on what we are hearing from customers, if we're not seeing recurrence of like a year or something, there will be more skepticism. So I think there is like a, it is time bound, if you will.

Alessio [00:34:15]: So you finally gave us some numbers on Stack Overflow usage. I think that's been a Twitter meme for a while, whether or not Chad GVT killed Stack Overflow. In the survey, 58 people said they used it less. And 94% of them said they used it less because of Copilot and Chad GVT, which I think it kind of makes sense. I know Stack Overflow tried to pull a whole thing. It's like, no, the traffic is going down because we changed the way we instrument our website. But I don't think anybody. And then you add right after that expectation of job impact by function and operations, people, 8 out of 10, basically, they think it's going to, it's going to really impact their job. Designers were the lowest one, 6.8 out of 10. But then all the examples you gave were designers of a job being impacted by AI. Do you think there's a bit of a dissonance maybe between like the human perception is like, oh, my job is like, can possibly be automated? It's funny that the operations people are like, yeah, it makes sense. I wish I could automate myself, you know, versus the designers or maybe they love their craft more. Yeah, I don't know if you have any thoughts on who will accept the first, you know, that they should just embrace the technology and change the way they work.

David [00:35:21]: Yeah, that's interesting. I think it's probably going to be engineering driven. I mean, I think you two are very well, maybe you two even started some of this wave and sort of the AI engineer wave. I think the companies that adopt AI the best, it is going to be engineering driven, I think, rather than like operations driven or anything else. And the reason for that is, I think the rise of this like profile with AI engineering, like AI is very philosophical, like AI is a tool in my head. Like it is not a, in my head, I think we're actually pretty far from AGI. But AI is not like a, you know, thing that it's not like a black box where like it does everything you want it to do. The models that we have today require like very specific prompting, for example, in order to get like, you know, really good results. And the reason for that is, it's a tool that, you know, you can use it a specific way. So if you use it the wrong way, it's not going to produce good results for you, actually. It's not like by itself taking a job away, right? And so I think actually, to adopt AI, it's probably going to be going to have to be engineering first, basically, where engineers are playing around with it, figuring out limitations of the models, figuring out like, oh, maybe like using vectorized databases is a lot better, for example, maybe like prompting in this particular way, it's going to be a lot better, etc. And that's not the kind of stuff that I think like an operations team is going to really be like experimenting with necessarily. I think it really has to be engineering led. And then I think the question is, well, what are the engineers going to focus on first? Like, are they going to focus on design first or like operations first? And that I think is more of a business decision. I think it's probably going to be more like, you know, the CEO, for example, says, hey, we're having trouble scaling this one function. So like, why don't we try using AI for that? And let's see what happens, for example. And so in our case, for example, we are really we have a lot of support issues. So what I mean by that is we have a really, really high performance support team. But we get a lot of tickets. And the reason for that is, you know, we're a very dynamic product, you can use it in so many different ways. And we'll have a lot of questions for us, basically. And so we were looking at, well, you know, can we, for example, draft some replies and support tickets, you know, by AI, for example, can we allow our support agents to be, you know, hopefully, you know, double as doubly productive as before, for example. So I guess I would say it's like business needs driven, but then engineering driven after that. So like, you know, we the business decides, okay, well, this is where AI can be most applied. And then we assign the project to an engineer, and the engineer goes and figures it out. I honestly am not sure if like the operation, we're gonna have much of a, like, if they accept or reject it, I don't know what's gonna change the outcome, if you will.

Alessio [00:37:40]: So another interesting part was the importance of AI in hiring. 45% of companies said they made their interviews more difficult in the in the engineering side, made interviews more difficult to compensate for people using copilot and chat GPT. As they change every tool, like, have you? Yeah, have you thought about it? I don't know how much you're still involved with engineering hiring, I get the company, but I'm curious how we're scaling the difficulty of interviews, even though the job is the

David [00:38:11]: same, right?

Alessio [00:38:11]: So just because you're gonna use AI doesn't mean the interview should be harder. But I guess it makes sense.

David [00:38:16]: Our sense, basically, the survey, and this is true, we believe, too, is we are most when we do engineering interviews, we are most interested in assessing like critical thinking or thinking, you know, on the spot. And I guess, you know, when you hire the employee, you know, in the end, the job of employees to be productive, which they choose whatever tools they want to be productive. So, you know, that's kind of our thinking, too. However, we do think that, you know, if you think about it from a first person's way, if your only method of like coding is literally copy pasting, you know, off of chat GPT, or like, you know, it's pressing tab and copilot, I think that would be concerning. And so, for that reason, we still do want to test for like, you know, fundamentals understanding of comp sci. Now, that said, I think if you're able to use chat GPT or copilot, let's say competently, we do view that as a plus, we don't view it as a minus. But if you only use copilot, and you aren't able to reason about like, you know, how to write a for loop, for example, or how to write fizzbuzz, that would be highly problematic. And so, for us, we do today is we'll base a screen share, or a rest is a hackpad, actually. So it's, sorry, this is no copilot there to sort of see what they're doing, or see what they're thinking. And we really want to test for thinking, basically. But yeah, I mean, we ourselves internally have embraced copilot, and we would encourage engineers to go over this copilot too. But we do want to test for understanding of what you're doing, rather than just copy pasting a copilot.

Alessio [00:39:27]: The other one was AI adoption rate, only 27% are in production. Of that 27%, 66% are internal use cases. Shout out to retool, you know, do you have a mental model as to how people are gonna make the jump from like, using it internally to externally? Obviously, there's like all these different things like privacy, you know, if an internal tool hallucinates, that's fine, because you're paying people to use it basically, versus if it hallucinates to your customer, there's a different bar. Because for you, if people build internal tool with retool, there are external customers to you, you know, so I think you're on the flip side of it.

David [00:40:02]: Yeah, I think it's hard to say, maybe a core retool belief was actually that most software built in the world is internal facing, actually, which actually sounds may sound kind of surprising, you know, for some of you hearing this, but effectively, like, you know, we all work at Silicon Valley, right? We all work at businesses, basically, that sell software as, you know, as sort of a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. If you look at like, you know, the clothing manufacturer that I was talking about, they're not a software company, like they don't sell software, you have to make money, they sell clothing to make money. And most companies in the world are not software companies, actually. And so most of the engineers in the world, in fact, don't work at Silicon Valley companies, they work outside of Silicon Valley, they work in these sort of more traditional companies. So if you look at the Fortune 100, for example, probably like 20 of them are software companies, you know, 480 of them are not software companies. That's the employable software engineers. And so most of the software engineers in the world, and most of the code engineers in the world actually goes towards these internal facing applications. And so, for all the reasons you said there, like, I think hallucination matters less, for example, because they have someone checking the output, and consumer, so hallucination is more okay, it's more acceptable as well. Yeah, it can be unreliable, because it's probabilistic, and that's also okay. So I think it's kind of hard to imagine AI being adopted in a consumer way without the consumer like opting in, like, Chachapiti is very obviously a consumer, the consumer knows that it's Chachapiti, they're using it. I don't know if it's going to make its way to like the banking app anytime soon. Maybe for like, even for support, it's hard. Because if it hallucinates, then, you know, it's actually quite bad for support if you're hallucinating, right? So it's, yeah, it's hard to say. I'm not sure.

Alessio [00:41:50]: Yeah, I think a lot of people, like you said, we all build software. So we expect that everybody else is building software for other people. But most people just want to use the software that we build out here. I think the last big bucket is like models breakdown. 80% of people use it, just use OpenAI. Some might experiment with smaller models. Any insights from your experience at Retool, like building some of the AI features? Have you guys thought about using open source models? Have you thought about fine tuning models for specific use cases? Or have you just found GPT-4 to just be great at most tasks?

David [00:42:24]: Yeah, so two things. One is that from a data privacy perspective, people are getting more and more okay with using a hosted model like a GPT-4, for example. Especially because GPT-4 or OpenAI often has to have enterprises who went to some companies already because I think a lot of CIOs are just like, let's get a second house. Like, you know, let's use Azure, for example. And, you know, let's make it available for employees to experiment with. So I do think there is more acceptance, if you will, today of feeding data into GPT. That's going to take some sensitive data. People might not want to do so. Like, you know, feeding in like earnings results data, you know, three days for you to announce earnings, like probably is a bad idea. You probably don't want people to be writing your like earnings statement for you. So yeah, there's still some challenges like that. But I think actually open source models could actually help solve like a lot of greed when it comes to, and that can be exciting. So that's maybe just one thought. The second thought is, I think OpenAI has been really quite smart with their pricing. And they've been pretty aggressive of like, let's get, you know, let's create this model and sell it at a pretty cheap price to make it such that there's no reason for you to use any other model. Just from like a strategy perspective, I don't know if that's going to work. And the reason for that is you have really well-funded players like Google or like Facebook, for example, that are actually quite interested. I think if it was creating startups, OpenAI would win for sure. Like at this point, OpenAI so far had from both a model and a pricing perspective that like there was no reason for it to go just really, I think, in my opinion, at least a startup model. But if like, you know, Facebook is not going to give up on AI, like Facebook is investing a lot in AI, in fact. And so competing against a large FANG company on making a model open source, I think that is challenging. Now, however, where we are right now is I think GPT-4 so far in terms of performance and I would say a model performance is so important right now because like the average, I'm not going to argue LLAMA-2 is actually so far behind, but like customers don't want to use LLAMA-2 because it's so far behind right now. And so that I think is part of the challenge. As AI progress slows down, so if we get like LLAMA-4 and LLAMA-5, for example, maybe it's a comparable at that point like GPT-5 or GPT-6, like it may get to the point where it's like, look, I just want to use LLAMA. Like it's safer for me to host it on-prem, it's just as fast, just as cheap, like why not basically? But I think right now we are in this state, we're opening up next year really well, I think. And right now they're thriving, but let's see what happens in the next year or two.

Swyx [00:44:40]: What are you going to ask differently for the next survey? Like what info do you really actually want to know that's going to change your worldview?

David [00:44:46]: I'll also ask you that, but if you have any ideas, let me know. For us, actually, we were planning on asking very similar questions because for us, the value of the survey is mostly seeing changes over time and understanding like, okay, wow, for example, GPT-4 Turbo MPS has declined. That would be interesting, actually. One thing that was actually pretty shocking to us was, let me find the exact number, but one change that we saw, for example, if you compare GPT-3.5 MPS, I want to say it was like 14 or something, it was not high, actually. The GPT-4 MPS thing was like 45 or something like that, so it was actually quite a bit higher. So I think that kind of progress over time is what we're most interested in seeing, is are models getting worse, models getting better? Are people still loving PG Vector? Do people still love Mongo? Stuff like that. That I think is the most interesting.

Swyx [00:45:33]: It seems like you're very language model focused. I think that there's an increasing interest in multi-modality in AI, and I don't really know how that is going to manifest. Obviously, GPT-4 Vision, as well as Gemini, both have multi-modal capabilities. There's a smaller subset of open source models that have multi-modal features as well. We just released an episode today talking about IdaFix from Hugging Face, and I would like to understand how people are adopting or adapting to the different modalities that are now coming online for them. What their demand is relative to, let's say, generative images versus just visual comprehension versus audio versus text-to-speech.

David [00:46:15]: What do they want?

Swyx [00:46:15]: What do they need? And what's the sort of forced, stacked ranked preference order? It's something that we are trying to actively understand because there's this sort of multi-modality world, but really multi-modality is kind of... I've been thinking about this phrase, multi-modality is like cancer. It's this umbrella term for actually a whole bunch of different things that aren't quite honestly not really that related to each other unless in the limit. But it tends towards maybe everything uses transformers and ultimately everything can be merged together with a text layer because text is the universal interface. But if you're given the choice between, I want to implement an audio feature versus I want to implement an image feature versus video, whatever, what are people needing the most? Should we pay the most attention to what is going to be the biggest market for builders to build it?

David [00:47:03]: I don't know.

Swyx [00:47:04]: I think I would just kind of zoom out a little bit to just a general founder questions. You have a lot of fans in the founder community. I think you're just generally well-known as a very straightforward, painstaking person about just business. Something that is the perception from Joseph is that you have been notably sales-led in the past. That's his perception. I actually never got that, but I'm not that close to your sales portion. And it's interesting to understand your market, the internal tooling market versus all the competition that's out there. There's a bunch of open source retools and there's a bunch of... I don't know how you categorize the various things out there, but effectively what he's seeing and what he's asking is, how do you manage between enterprise versus ubiquity? Or in other words, enterprise versus bottom-up, right? I was actually surprised when he told me to ask that question, because I had always assumed that you were a self-serve, sign-up, bottom-up led. But it seems like you have a counter consensus view on that.

David [00:48:04]: Yeah. So actually when Retwelf first started, we started mostly by doing sales, actually. And the reason we started by doing sales was mostly because we weren't sure whether we had product-market fit and sales seemed to be the best way of proving whether we had product-market fit out. Because I think this is true of a lot of AI projects. You can launch a project and people might use it a bit and people might stop using it and you're like, well, I don't know. Is that product-market fit? Is that not? It's hard to say, actually. However, if you work very closely with the customer in a sales-led way, it's easier to understand their requests, understand their needs, and stuff like that, and actually go build a product that actually serves them really well. And so basically, we viewed sales as like working with customers, basically, which is like, I think actually quite a, I think it's a better way to describe it, what sales is of an early-stage company. And so we did a lot of that, certainly, when we got started. I think we, over the last maybe five years, maybe like three years ago, four years ago, something like that, I think we have invested more on the self-serve ubiquity side. And the reason for that is when we started Retwelf, we always wanted, actually, some percent of software to get built inside of Retwelf, whether AI software or origin software or broadly UIs and whatnot, but like software, basically. And for us, we're like, we think that maybe one day, 10% of all the code in the world could be written inside of Retwelf, actually, or 10% of the software could be running on Retwelf, which would be really, really cool. And for us to achieve that vision, it really does require a broad-based option of the platform. It can't just be like, oh, only like 1,000 customers, but the largest 1,000 companies in the world use it. It has to be like all the developers in the world use it. And for us, there's like, well, I think 25, 30 million developers in the world. That's of course, how do you get to all the developers? And the only way to get to those developers is not by sales. You can't have a salesperson talk to 30 million people. It has to be basically in this sort of moms-up, product-led, Ubiquity kind of way, basically. And so for us, we actually changed our focus to be Ubiquity, actually, last year. So our gold star metric used to always be sort of revenue-generated or revenue-generated. We actually changed it to be number of developers building on the platform, actually, last year. And that, I think, was actually a really clarifying change because obviously, revenue was important. It funds a lot of our product and funds the business. But we're going to fail if we aren't able to get to something like 10, 20, 30 million developers one day. We can't convince all developers that Retool's a better way to build a sort of class of software, let's say, internal applications for today. And so I think that has been a pretty good outcome. I think about the last five years of Retool. I think the starting off with sales, so you can build revenue, and then you can actually build traction, and you can hire more slowly. I think it was really good. I do think the focus towards bottoms-up Ubiquity also was really important because it helps us get to our long-term outcome. What's interesting, I think, is that long-term Ubiquity actually is harder for us to achieve outside of Silicon Valley. To your point, I think at Silicon Valley, Retool is reasonable Ubiquitous. I think if you're starting a startup today and you're looking to build an internal UI, you're probably going to consider Retool, at least. Maybe you don't choose it because you're like, I'm not ready for it yet or something. But you're going to consider it, at least. And when you want to build it, I think it's actually a high probability you will actually end up choosing it. It's awesome. But it's that if you think about a random developer working at, let's say, like an Amazon, for example. Today at Amazon, actually, we have, I think, 11 separate business units that use Retool at this point, which is really awesome. So Amazon is actually a big Retool customer. But the average here at Amazon probably has never heard of Retool, actually. And so that is where the challenge really is. How do we get, like, I don't know, let's say 10,000 developers at Amazon building via Retool? And that, again, I think is still a bottom-up ubiquity thing. I don't think that's like a, I don't think we're going to like, you know, go to Amazon and knock on every developer's door or send out an email to every developer and be like, go use Retool. They're going to ignore us, actually. I think it has to be, use the product, and you love it, you tell your co-worker about it. And so for us, a big bottom-up ubiquity, but marrying that with enterprise or the community business has been something that's really near and dear to our hearts.

Swyx [00:51:54]: Yeah, just like general market thoughts on AI. Do you spend a lot of time thinking about like AGI stuff or regulation or safety What interests you most, you know, outside of the Retool context?

David [00:52:07]: There's a lot of hype in AI right now. And it's again, not too many use cases. So for us, at least from a Retool context, it really is, how do we bring AI and have it actually meet business problems? And again, it's actually pretty hard. Like I think most founders that I've met in the AI space are always looking for use cases, never have enough use cases, right? Sort of real use cases, people pay money for them. But I think really where the Retool interest comes from, me personally, I think philosophically, yeah, I've been thinking recently myself a bit about sort of intentionality and AGI and like, you know, what would it take for me to say, yes, you know, GPT-X for, you know, any sort of model actually is AGI. I think it's kind of challenging because it's like, I think if you look at like evolution, for example, like humans have been programmed to do like three things, if you will, like, you know, we are here to survive, you know, we're here to reproduce and we're here to like, you know, maybe this is just two things, I suppose. So basically, to survive, you have to go eat food, you know, for example. To survive, maybe like having more resources helps you want to go make money, you know, for example. To reproduce, you should go date, you know, or whatever, you get married and stuff like that, right? So like, that's, we have a program to do that. And humans that are good at that have propagated. And some humans that, you know, we're not actually surviving, probably have disappeared just due to natural selection. Humans that we're not interested in producing also disappeared because there are less of them, you could say, because they just, they just stopped carrying on basically. And so, so it almost feels like humans have sort of naturally self-selected for these like two aims. I think the third aim I was thinking about was like, does it matter to be happy? Like, maybe it does. So maybe like happier humans, you know, survival, it's hard to say. So I'm not sure. But if you think about that, and they're all just like AIs, if you will, right now, we're not really selecting AIs for like, you know, reproduction. Like, it's not like, you know, we're being like, hey, AI, you know, you should go make 30 other AIs. And you know, those that make the most AIs, you know, are the ones that survive. We're not saying that. So it's kind of interesting sort of thinking about where intentionality for humans come from. And like, I think you can argue the intentionality of the human space that comes out of these three things. You know, like, if you want to be happy, you want to survive, you want to reproduce. That's like basically your sort of goal, you know, in life. Whereas like, the AI doesn't really have that. But maybe you could program it in. Like, if you, you know, prompt inject, for example, like, hey, AI, you know, go do these three things. And you can even create a simulation, if you will, like all these AIs, you know, in the world, for example. And maybe you don't have AGI in the world, which I think is kind of interesting. So that's kind of stuff I've been thinking about when I talk about with some of my friends from a sort of philosophical perspective. But yeah, it's kind of interesting.

Swyx [00:54:29]: Yeah, my quick response to that is we're kind of doing that. Maybe not at the sort of trained final model level, but at least at the data sets level, there's a lot of knowledge being transferred from model to model. And if you want to think about that sort of evolutionary selection pressure, it is happening in there. And, you know, I guess one of the early concerns about being in Sydney and sort of like bootstrap self bootstrapping AGI is that it actually exists if these models are sentient, it actually exists in their incentive to get as much of their data out there into our data sets so that they can bootstrap themselves in the next version as they get trained. That is a scary sobering that we need to try to be on top of.

Alessio [00:55:13]: David, I know we're both fan of Hofstadter's GB and actually saw in one of your posts on the Segovia blog, you referred to the anteater. I don't even know if you call them chapters and GB is just kind of like this, this continuous rift. But basically, like our ants are like not intelligence, but like ant colony has signs of intelligence. And I think Hofstadter then use that to say, hey, you know, neurons are kind of like similar and then computers maybe will be the same. I've always been curious if like we're drawing the wrong conclusion from like neural networks where people like, oh, each weight is like a neuron and then you tie them together should be like a brain. But maybe like the neuron is like different models that then get tied together to make

David [00:55:57]: the brain.

Alessio [00:55:57]: You know, we're kind of looking at the wrong level of abstraction. Yeah, I think there's a lot of interesting philosophical discussions to have. Sean and I recorded a monthly recap podcast yesterday, and we had a similar discussion on are we using the wrong? What did you say Sean on the plane and the bird? I think that was a good analogy.

Swyx [00:56:16]: The sour lesson, are we using the wrong analogies? Because we're trying to be inspired by human evolution and human development, and we are trying to apply that analogy strictly to machines. But in every example in history, machines have always evolved differently than humans. So why should we expect AI to be any different?

David [00:56:33]: Yeah, it is interesting because it does feel like, yeah, if you sort of peer under the hood of AGI, if you insist that AGI, we'd have always used AGI for things like a human. And that is the Turing test, I suppose. But whether that is a good point, like if it works, no, it's not the Turing test. The Turing test is if the output is the same as a human, then I'm happy. I don't really care about what's going on inside. And so it feels like caring about the inside is like a pretty high bar. Like, why do you care? It's kind of like the plane thing for flies. It's not a bird. I agree. It does not fly necessarily the same way as a bird. Physically, it does, I suppose. But you see what I mean? It's not the same under the hood. But it's OK for the flies. That's what I care about. And it does seem to be like AGI probably doesn't think and can achieve outcomes that I give it. It can achieve its own outcomes. And if it can do that, I kind of don't care what it is under the hood. It may not need to be human life at all. It doesn't matter to me. So I agree. Awesome.

Alessio [00:57:26]: No, we kept you long. Actually, I have GUB right here on my bookshelf. Sometimes I pick it up and I'm like, man, I can't believe I got through it once.

David [00:57:34]: It's quite the piece of work. It's a lot of fun, though. Yeah.

Alessio [00:57:38]: I mean, I started studying physics in undergrad. So, you know, it's one of the edgy things that every physicist starts going through. But thank you so much for your time, David. This was a lot of fun. And looking forward to the 2024 state of AI results to see how things change.

David [00:57:54]: Yeah, I'll let you know. Thanks, both.