Transcript
[00:00:00] Victoria Melnikova: Hi everyone. This is Dev Propulsion Labs, our podcast about the business of developer tools, and my name is Victoria Melnikova. I’m head of new business at Evil Martians, and I’m very excited to meet today’s guest, Jeff Huber, co-founder and CEO at Chroma. Hi Jeff.
[00:00:21] Jeff Huber: Hi, how are you?
[00:00:22] Victoria Melnikova: I’m great. How are you doing?
[00:00:23] Jeff Huber: Wonderful, thanks.
[00:00:25] Victoria Melnikova: I’m actually very excited to record today’s episode because I’m hosting you, but you’re hosting me. We’re at Chroma’s office here in the heart of San Francisco, and I’m very excited to pick Jeff’s brain about what’s going on with Chroma. So who is Jeff?
[00:00:43] Jeff Huber: Well, I’ve worked now for something like 10 years in applied machine learning and developer tools. And you know, this is back before we called it AI and certainly not gen AI. It was really just deep learning. And I, you know, I think I’ve always wanted to find places to build really great tools and I love the leverage that building really great tools enables, and obviously open source in particular is even like a further accelerant of that.
And so, you know, one kind of theory of mind, I guess, you know, not laying down—you know, this is not a psychiatric session, but a theory of mind of Jeff might triangulate on those things and kind of just to connect that to the Chroma story a little bit. You know, Chroma started basically because I had worked in applied machine learning for 10 years. And just seeing how challenging it was to build production systems with AI and machine learning and, you know, these systems seem to hold a lot of promise, you know, could or should they be made reliable enough to be trusted. But like, that was always just such a really challenging thing to figure out and still is today obviously. And, you know, I think kind of classic “there must be a better way” sort of thesis. And so that was one part of it. And then the other part of it was that at the time, at least that we started, you know, nobody really knew what latent space was. You know, embeddings were not really a constant, that very popularized. And so, you know, we had seen some research and we kind of had an intuition that viewing data in high dimensionality, the way that models see their own data.
Had a lot of promise for interpretability, being able to understand how and why these systems are doing certain things. And also, you know, therefore also had a lot of promise to being able to sort of deterministically iterate and improve these systems as well. And so, you know, this is very much a hunch. We didn’t necessarily have evidence, direct evidence before starting the company that it was going to work. And then of course the world kind of shifted something like six months—into the company. And now of course saying like embeddings are good or valuable or useful sounds hilariously obvious. But at the time I think, you know, the vast majority of VCs that we sort of pitched and you know, said this to, their eyes just glazed over and they had no idea what we were talking about. And so yeah—
[00:02:59] Victoria Melnikova: You’re kind of reshaping what’s going on. Right. I was just on a call with Saoud from Cline talking about how RAG is dead. So like what is the premise of Chroma? And we can talk a bit more about Chroma, like what it offers to, you know, users. Yeah. I mean, how are you shifting the narrative basically?
[00:03:19] Jeff Huber: Yeah. I mean, I think like, you know, RAG is—the king is dead, long live the king. You know, the reports of my death have been greatly exaggerated. Like all these phrases apply so well to RAG. You know, the irony is that, you know, the Cline team seems to have found—and I will, you know, I give them crap on the internet for this. I’m happy to, you know, give them crap on the record here as well. You know, they say they don’t use RAG and yet they literally give their agents the ability to regex and grep the repo and like last I checked that is retrieval augmented generation. Yes, it is not like single dense vector search, but like nobody’s ever said that that is a panacea and that’s kind of like a false boogeyman in my opinion. I do think that the narrative has shifted in the last few months from mm-hmm, like RAG to context engineering, which I’ve both been a part of proliferating that concept and like I think is a very positive shift. RAG, like, never meant anything to anybody, and as a result meant everything to everybody as well. And like it’s just overall a poor phrase to help shape the actual job to be done versus context engineering is a great phrase and implies the existence of a context engineer, which is literally the job of most AI teams today. Like your job is not to fine tune the model. Your job is to figure out how to engineer the context windows such that the model could be successful at the job you need it to be good at. And so it’s like a very positive change. Now, search and retrieval is, you know, an extremely ubiquitous and important tool in solving the challenges of context engineering. But yeah, I think RAG overall, I mean, you know, most of what you see on Twitter is like brain rot and you know, just sort of bad for the pod, bad for your soul and hype. And so maybe we should all log off and touch grass more. I think, you know, anybody who has very extreme claims, that also demands extreme evidence and I have never seen any evidence of any of these claims. So.
[00:05:08] Victoria Melnikova: So let’s pause here for a second. When we talk about context engineering, I mean, you and I, we kind of exist in the bubble of the Bay Area, right? All of those things make a lot of sense. But if we talk to an average engineer globally, what does context engineering mean?
[00:05:22] Jeff Huber: Yeah, so really one way to think about what LLMs are—you know some people, you know, mainly the large language model labs will—they want you to believe in artificial general intelligence and super intelligence. They want you to believe that AI is this literal deus ex machina. It is a God that is going to come out of the machine. It’s going to solve all of humanity’s problems. And listen, if I had to raise a hundred billion dollars, I would absolutely be telling you the same thing. I think that that is not actually what AI is though, and AI is really just a new era of computing. That’s a new way of building software. Classic computers are very good at running structured programs with structured inputs and outputs. And LLMs are very good at running unstructured programs, unstructured inputs and outputs. And you know, then you sort of ask yourself the question, well, you know, you sort of pass code to a CPU to run a program, kind of classic computing. And in AI you pass a context window to an LLM or to a vision model to be processed. And so, you know, you think about the context window—that is the tool that is your programming canvas. And you know, if you put bad stuff in there, the model will get distracted or get overwhelmed or get it wrong. If you put good stuff in there, the model has a fighting chance to get it right. And so the job then becomes figuring out again, like what goes into the context. Because that is how you sort of set the behavior and you program the system and you know, this problem is not as easy as it may at first appear. You know, we released a technical report a few months back that has kind of gone far and wide on the internet, on the topic of context rot, which is looking at how both the ability of models to pay attention to and the ability of models to reason over their inputs degrades as the sort of context window increases in size. And so, you know, models might be marketed as having like a million tokens of context and yet, you know, real builders who are building them to do things that are difficult, you know, you usually don’t trust them once they pass like 40,000 tokens or something like this. And so, you know, there’s a bunch of techniques then you can employ to work around these constraints. But this is the job of context engineering. You know, you’re a smart person, maybe you have the tools of programming available to you and you can vibe code the rest. You need to figure out, basically, again, like how do I curate the right mixture of information mm-hmm such that the model has a very good chance of producing the outcome that I want. And doing so reliably, right? Not just doing so one time, but handling, you know, the real world’s a pretty messy place. Unstructured data has a tendency to sort of, you know, mug systems if you will. And you get mugged by reality. And reality is fully complex. And so, this is also, I think, part of the challenge of context engineering is like, how do I build systems that are sort of very reliable at the limit and aren’t just, you know, reliable seven times out of 10. Reliable 99.999999 nine times out of 10. So—
[00:08:09] Victoria Melnikova: So Chroma’s positioning, and I want to read it because I don’t want to butcher it, is an AI native, open source embedded database. What does it mean in 2025?
[00:08:18] Jeff Huber: The phrasing that I would use maybe today is—I see very much we’re working on is building modern search infrastructure. For AI, modern is mostly in contrast to legacy. Search infrastructure, which is very costly, very difficult to scale, very difficult to run. Doesn’t have a lot of advanced features that you would want for AI versus a modern distributed system. Which is effortless to scale. Effortless to run, very cost effective, very fast. And then the “for AI” part has many different facets, but I’ll give you kind of one conjecture. You know, today it may still be the case that human queries, mm-hmm, sort of, you know, outnumber AI queries. Yeah. But you can imagine in two years that’s going to be a million to one on the side of AI. And so, you know, AI is by far the most important user of search. Both obviously submitting search queries, but also digesting search results. Yeah. And I think that also motivates and necessitates a lot of things at the infrastructure layer to support that large secular shift in how search works. Search and retrieval work. And so that’s the problem that we’re working on. Yeah. Is really building a whole suite of both data tools, compute tools, platform tools around enabling teams to solve those kinds of problems.
[00:09:41] Victoria Melnikova: So you founded it a few years ago, right? In 2022. If I’m not mistaken. And since then, has your positioning shifted a lot or—let’s say later in 2024, 2025. Do you feel like you had to adjust the positioning a lot because of this upcoming wave?
[00:09:58] Jeff Huber: My disposition is always, I think that the best companies are quite fixed on the long term future. So like where they want to be in 10 years. Like rarely changes, like, you know, maybe occasionally something massive happens that totally changes where the company should be in 10 years, but most of the time you can kind of actually see 10 years forward—it’s not that hard. Also, seeing like a month into the future, it’s not that hard. You kind of have all the information. You know, I think that the death zone for predictions is generally in the two to three year period. Because it’s sort of hard to know, you know, again, you know where you are, you know you’re gonna get there. But the path to get there is not linear. Yeah. You don’t know. Right. And, you know, you might need to go that way and then that way. And so it kind of depends on the market and, I think, you know, the founders that also are very—let’s not use the word reactive, but are like sensitive mm-hmm to the market and are only wanting to work on things and prioritizing commercialized things that their users really care about. Yeah. Like that’s ultimately the goal—to grow fast you have to do things people care about. And so it’s fine to build a feature that’s really important in 10 years, but if nobody cares about it today then you sort of just wasted your time. Yeah. Unless it’s gonna take you 10 years to build it. And so I think we’re very willing to, you know, sort of massage the positioning to mm-hmm reach our ICP in a way that they understand and in a way that they’re excited about, but I don’t think it’s at the expense of the long term vision.
[00:11:23] Victoria Melnikova: This is so interesting and actually the 10 year mark is coming up, right? It’s like in five, six years at this point. So what is Chroma in five years?
[00:11:33] Jeff Huber: Again, going back to the founding of the company. The goal of Chroma has from the start been to enable developers to somewhat deterministically and easily build production systems. And I think that comes through continual learning. Ultimately, this is the goal of, I think, building very reliable AI systems—ones that deal very well at the edges of their domain knowledge, they have to deal well with the edge cases. They have to build an intuition for how to deal with edge cases. You know, you and I can walk into a new, you know, job or environment and somebody can teach us how to do something very quickly actually. And then we can generalize and pattern match and apply that to new scenarios. And, you know, maybe we do something wrong and the person sitting next to us says, “Hey, next time do it sort of like this”—this apprenticeship model. And we can kind of build up a corpus of tacit knowledge in our brains. And then we’re actually good at that job and we do get to that 99.99999% reliability that is so desirable and then enables other people to trust us and everything else. And so we want our machine systems to have that same ability to get better over time. People call this continual learning. And, you know, what tools do you have available to you for an AI system to make it better over time? You can change the prompt. Like the system prompt. You can change the weights of, you know, one or many models, you know, kind of in the stack. And then you can change the data the model has access to. And likely it’s the case that all of these things you want to change to get the most, you know, reliability. I’d say that most of our customers today, you know, are not yet in the business of fine tuning the models. Yeah. I think that was kind of not a great idea for many years. It does feel like that’s maybe starting to change actually. Prompt optimization has become much, much more popular, which is really exciting. You see projects like DSPy, popularizing methods like MIPRO, where you can do automated prompt optimization. Which is really awesome. And then obviously a lot of our customers both use Chroma for just traditional static RAG or kind of knowledge bases, corpus, et cetera. But increasingly, we’re starting to see customers use Chroma also for memory broadly. And like memory, I don’t love that word, but broadly what memory means is, the intuition here is you want the system to continually get better over time. And that could be either getting better at a given task and obviously there’s one—well, I guess always a given task. One such task is user personalization. Yeah. And so it could be, you know, the model’s getting better at talking to you, the model is getting better at doing this kind of task or both. And so that’s, I would say, a fast growing use case. Kind of in our customer set is memory broadly. And so again, this is all in our minds in the direction of continual learning. And I hope that in five years we’ve just fully solved that problem for the majority of business use cases.
[00:14:14] Victoria Melnikova: Interesting. I want to talk about business for a little bit. So I think I saw in one of your interviews that one of the early hires was a designer. Am I wrong in that? So it’s not common for databases to pay so much attention to design and finesse, and if one goes to your website, they will see that there’s a lot of attention to detail and it’s very polished. It’s not common in developer tools now. I think it’s becoming more common, but— —it’s admirable. So for you in making that decision, what was the thinking, how do you justify that kind of investment early on? Considering that ROI from that is not apparent. Basically take us through your present there.
[00:15:02] Jeff Huber: I mean, I probably would want to invest in design even if it didn’t change the bottom line. Just because I have to live a certain way and I couldn’t sleep at night if it all sucked, you know? You know, there’s—and to be clear, like brutalist design is still design, right? And so, you know, a webpage with only Times Roman 12 point font, black text and a white background, that’s still design. So that’s not—I’m not saying that that’s anti-design, right. But if it looked ugly, I just could not live with myself. And so it has to sort of, for my own, you know, estimation, be beautiful. Broadly speaking, I would say engineers and developers tend to have more of a blind spot when it comes to the importance of brand. And obviously the opposite of this is, you know, consumer packaged goods companies in New York or LA. Right? Like they’re a hundred percent brand and then it’s just like a product they buy off Alibaba, right? It’s like all brand. But I will say that in practice, the number of people that have a positive association with Chroma and say to me, “I love the design of your website.” Yeah. Like, is many and that feels accretive and long-term valuable and important. And so I think in that way, for developer tools companies, design can be like a bit of a moat as well because I mean, brand is, you know, a very useful thing to build. If you need great design to build brand, then you also need to be able to have a good design culture. You need to be able to hire great designers, you need to be able to manage and lead great designers and that is sort of a difficult thing to do very, very well. And so, you know, anything that you can do very well that your users care about that is difficult to do is something that, you know, increases the chances of your project being more successful. Obviously for open source, you know, commercial open source, success is both defined by the number of customers or users who are just using your product out there in the world, and then jointly the number of customers that are using, you know, your advanced services or whatever the revenue model is. And I think that any, you know, commercial open source founder or even just open source project maintainer that really cares about impact—you know, more users is better in a way, right? Like more users means that your project is having more of an impact on the world. And I think it’s something that we all, you know, desire deep down is to have an impact on the world.
[00:17:34] Victoria Melnikova: This is very exciting to hear, and I’m reflecting on this kind of level of perfectionism and attention to detail that you yourself portray. When it comes to hiring team members, is that something that you’re actively looking for in people that you would hire at Chroma?
[00:17:51] Jeff Huber: We have a pretty well honed kind of hiring process, and I think it’s something like for every hundred phone screens we have, we have one accepted offer. Like sort of top of funnel to bottom of funnel. It would be bad to just sort of, you know, throw lots of low quality candidates into the pipeline and make that number even more dramatic, obviously. So there’s like a moral hazard here to these sorts of ratios, if you will. But, and that’s not because everybody that we interview, you know, isn’t, you know, really smart or really good at what they do. A lot of times it’s also just not a cultural fit. And I think we really have a culture of people who do sweat the details and they can just tell you, like right now some of the things that we’re working on—in private beta right now or releasing soon, like the amount of time that we put into API design is like I have not seen anywhere else. Maybe other people do this, but we will have all-team meetings. Which, you know, as far as meetings go are very expensive. Yeah. For like an hour or an hour and a half.
[00:18:53] Jeff Huber: Where everybody comes into a room. We all look at the API, we all nitpick the API, we all use the API, we try to find all the sharp edges with the API. Yeah. We just—we are obsessed with making the very best APIs possible. You know, getter names will have like 18 different variants. You know, like what is the actual collection.name or you know, client.name, right? Like we will obsess over the name and, you know, naming is the hardest problem in computer science. And so, you know, if that’s the case then APIs, which are sort of like a bundle or a basket of names, you know, by definition is combinatorially more difficult. But I think it’s incredibly important. Like you both want to make your API very approachable for new users. Naturally. And then, you know, maybe increasingly you want to make your API approachable for new AIs.
[00:19:44] Victoria Melnikova: There’s a lot of really great insight that you’re sharing. And I want to share this hot take—this level of perfectionism and this level of attention to detail sometimes could mean toxicity Hmm within the company culture. How do you make sure that that’s nicely balanced? Because you know, if you set yourself a goal of finding imperfections, you’ll find them. Right? Yeah. So where does that stop and how do you make sure that the culture at Chroma is still thriving and healthy? And not toxic?
[00:20:22] Jeff Huber: I think toxic cultures really only evolve when people have the strong opinion that they are right. That’s different than having the strong opinion that we should find the right answer. And so we try to hire people and we succeed at hiring people that have relatively low egos. Like for their own ideas. And, you know, they’re really invested in the pursuit of truth and finding the very best possible, you know, solution to a given problem or name. And, you know, one of our values is a phrase we use sometimes called “look for surprises.” And I think that disposition of, you know, I’m in this meeting not to prove myself correct. But I’m in this meeting with the disposition of, I am open and ready and willing to be surprised by possibly being wrong in some way, or not having perfect information—that is a value which helps with this. I’ll also say that I think that anything great is ultimately built by a small team of opinionated people. And so I don’t—I do not think that you can build something great by a large team.
[00:21:30] Jeff Huber: I do not think—and obviously large companies can be composed of smaller teams, so it’s not about the company size. I do not think you can build something great by being a large team. I do not think you can build something great by having unopinionated people. And so, you know, you sort of just, I think the only, again, the only way to do this is this way, and you see this in a lot of open source projects, right? Yeah. Like, you know, there’s many—you can just reference, you know, Linux itself, right? You know, it is maintained by a small group of highly opinionated people. And this is why it is so great and if it was like one large decision by committee, it would be a train wreck. And so, you know, I think that the desire for consensus whilst intuitively egalitarian and, you know, sort of feels good. Actually is like a death blow for many projects.
[00:22:14] Victoria Melnikova: I mean, it sounds almost like a controversy to be opinionated and to be excited about being wrong in a way. You know, because it’s like you open yourself up for discovery and learning about other opportunities, which could be better than what you had in mind originally. But it’s an interesting take, and I actually invite our listeners to take this as a takeaway from this conversation, because I think it’s liberating in a way to think about projects like that.
[00:22:42] Jeff Huber: Yeah. It makes sense that it would, you know, maybe on a rational basis it’s hard to do both at once. I agree. Yeah. The benefit is that humans are not very rational and so, you know, I think we can actually embody this superposition or the synthesis of both being opinionated about, “I think this is the best way.” And also being open to, you know, other ways of other ways.
[00:23:01] Victoria Melnikova: So when you founded Chroma, you had already had experience building a startup from scratch, right? And also growing inside Imgix from engineer to head of product, right?
Were there some lessons that you were surprised to learn when you founded Chroma? Like was there something like a revelation that you found, especially in the early days, you know, especially when you shared that there were some skeptics about vector databases and people didn’t quite understand what it means. And perhaps retaining that certainty in the product was hard. Yeah. Were there any lessons that came up?
[00:23:44] Jeff Huber: I mean, yeah. I’ve been working kind of as a part of various startups for some time and so I think that, you know, I brought obviously a lot of mistakes that I’ve made in the past into this company and I think as a result we’re able to also avoid a bunch of mistakes, which is good. We got both, I would say lucky, but also we were kind of in the right time, right place, right team, kind of in the early days of mm-hmm search and retrieval and RAG and everything else. Yeah. And you know, as a result, the project just in the earliest days grew like wildfire and, you know, it’s now used by tons of teams. I was just on a trip to New York meeting with a bunch of large financial institutions and you know, they’re just telling me things like, “Yeah, we have, you know, thousands of engineers that use and love Chroma” and, you know, this is kind of blowing my mind honestly. Yeah. Like even just last week. And so, you know, it’s obviously downstream of catching fire in the earliest days. And I—that’s not, you know, that’s not the destiny for all obviously open source projects. Some open source projects take, you know, many years to kind of reach that breakout period. But I would say that if something surprised me, it was, I think I had known that such a thing existed.
[00:24:54] Jeff Huber: —theory, but the feeling of that thing was very cool and different and new.
[00:25:01] Victoria Melnikova: Let’s talk about open source for a bit. Was it tough for you to understand what part of the project should be open sourced or how did you, in general—what was your strategy about the open source component of your product?
[00:25:16] Jeff Huber: I think for data infrastructure, it’s usually pretty straightforward, at least for OLTP-shaped data infrastructure, which search is, yeah, like an OLTP-shaped workload. You know, OLAP maybe is different. We can always talk about that if people are interested, but for OLTP—it kind of just has to be open source. Yeah. To be very successful, developers have to start on their local computers. Large enterprises have to have a continuity story and the security story of it being open source. And there’s just kind of no way around it. And you’re backing, you know, a live application. And so it’s not like an OLAP database where, oh, the query was slow, whatever, you know, or, oh no, the query failed, retry it. Yeah. You know, like a live database for backing real end users—it just has to work and it has to be very reliable. And so we kind of always knew that it should be open source. It shouldn’t be fake open source, so it shouldn’t be some weird license that has a bunch of hair on it. You know, it should be sort of MIT or Apache 2 or BSD 2 or 3-Clause and we went with Apache 2.
[00:26:18] Jeff Huber: And then we also believe that, you know, other people do weird stuff where they’re like, oh, the data plane is open source, but the control plane is closed source. But you don’t find that out till later. Gotcha. You know, and as you want to go beyond a single node distributed—well sorry, you gotta pay us money now. And I don’t know, I just think that that’s a short term optimization over a long term optimization. Yeah. And I think, I’ve always wanted to build the biggest company in the long term and that means playing for the long term. And so Chroma is a hundred percent open source. The database is all Apache 2, it’s all in our monorepo on GitHub. You can see everything. There’s not like a secret indexing algorithm that we’ve not open sourced or anything like this. It’s open source, and again, that’s not true for a lot of our competitors. So that being said, I don’t think it makes sense for everything to always be open source. Open source is a large commitment to your community. Yeah. You are—I see open source as making a real commitment to your community that you will be a good steward of this project. Yeah. You’ll take in their feedback, you’ll address their bugs even when these users are not paying you any money, right? Like you will continue to improve the project, keep it up to date, handle security, all of these things. And so in some ways, you do want to be thoughtful about the surface area of things that you open source. Yeah. And you know, not everything should be open source as a result. You also want to be able to move quickly at certain times, and, you know, if you have a database, well you can’t really move that quickly. You have to think about backwards compatibility and migrations and make sure that, like, that’s easy for users as well. And so, long story short, we’ve always been committed to—the engine, that is, the database is open source, it’s Apache 2, again, there’s no gotchas, no tricks. That being said, I think that we are okay with the idea of the car being commercial. The car sits around the engine and so, you know, I think a lot of people just want cars. Yeah. Like they want to get in the car, turn the key, go from point A to point B. They don’t have to think about the engine. They want to bring other people with them in the car and collaborate. They want to turn on the radio and the AC and the heated seats and, you know, want the accoutrements and workflows and, you know, obviously in practice this is, you know, auditing and security and auth and all this stuff, right? Yeah. Seeing their data and editing their data and all this stuff and, you know, in our mind that was a good thing. That was a thing that we thought made sense to not make open source. Again, not ‘cause we’re being overly greedy with it. Maybe we will open source it someday, but we needed, ultimately our team needed velocity on that thing. Yeah. And if you’re open sourcing it, you’re opening yourself up to the community giving input, which is great. But also in practice, it’s gonna make the thing go slower. And so we needed a lot of velocity. Anyways, this is how we thought about it. This analogy of engine and car. We’ve been extremely clear about from day one, both with our user base and with our team and I think there’s never ever been a problem at all. And you know, it’s—this is on our about page and will be on our about page forever. We are committed to making Chroma the ubiquitous open source solution in this category and I think you have—you can’t half ass it. You know? You have to stay true to that.
[00:29:22] Victoria Melnikova: Do you find that your open source users and your car users are the same people?
[00:29:27] Jeff Huber: Yeah. That’s the interesting part about what we’re doing as well is that they are extremely overlapping. And, you know, I would say for commercial open source, there’s monetizing data, monetizing compute. And monetizing config. And monetizing config and compute—you can start to have some weird moral hazard that happens. And it’s not always super aligned with your community and mm-hmm it can just be awkward, you know? Yeah. I think what’s interesting about monetizing data is our users from day one have asked us for a fully hosted, managed service mm-hmm from day one. And it took us a while to build it because we believed in building it the right way. But now you know, that is live. It’s growing very fast. And users just love it. Which is awesome. And, you know, we’re not forcing you to use our service. Right? Yeah. You can run distributed Chroma yourself. You can use single node Chroma yourself. Increasingly, we actually are running Chroma because it’s as simple as a Rust binary. We’re also putting it inside of Swift to run on iPhones and putting it inside of Android to run inside of, you know, Android devices and we’re working on maybe in the next month or two releasing it so you can run it in the browser? Yeah. Or in a Node process. And just anywhere, right? Like anywhere that you want to search and retrieve, you should be able to do so. Across edge devices, you know, serverless functions, you know, single node servers or distributed clusters. And, you know, our belief is that most users will benefit from not managing their own infrastructure. And you know, the very best infrastructure—who runs the infrastructure the best? It’s the people that, you know, design the infrastructure.
[00:31:06] Victoria Melnikova: So at this stage of the product where you have already found your product market fit, right, things are going smoothly, how do you prioritize your feature list, or how do you think about your roadmap? Mm. You obviously have your 10 year mission in mind. Yeah. But those stepping stones are ever changing. And who plays into making those decisions?
[00:31:26] Jeff Huber: You know, one way to think about the decision matrix is for a database like Chroma, people are choosing across a variety of factors. And this could be speed, cost, reliability, operational complexity, scalability, and accuracy. To name a few. So, you know, we naturally think about, okay, are we already best in class on speed? Are we best in class on cost, obviously. Those two are tightly related. You can usually spend more money to go faster, but people don’t always want to spend more money to go faster. So some sort of, you know, Pareto optimal point for, yeah, you know, the speed to cost ratio. For things like operational reliability and stability, this is something we’ve invested a ton in and, I think I’m very proud to say that, versus maybe the search infrastructure that came before Chroma. Chroma is truly zero config, zero knobs to tune. It just works. And people love that. And we’ve had stories from customers where their launch goes really well, and their Postgres falls over and their compute orchestration framework falls over. But Chroma just keeps, you know, chugging away. No problems at all. And obviously that makes us really happy to hear, but we designed it to do that. And so that wasn’t—that’s not an accident, right? That’s not—it’s not. It’s not unintentional. So I think this is the frame that we use or one of the things we use. You know, I also think that we view the job to be done as not—and this is maybe a bit of a heterodox take, but I think the job to be done is not quote “building a database.” Actually, I mean, that’s what people think that they want to buy. They want to buy a database or they want to buy data infrastructure and that’s fine. I’m happy to use the terms people want.
I think actually deep down, developers want to solve a certain type of problem and it just so happens that for this type of problem, you need to build it, you need to test it. You need to deploy it, you also need to monitor it, and then you also need to iterate on it. And it ends up feeling a little bit more like software in a sense, because again, the data that’s being retrieved is the programming of your context window as we discussed. And so it is kind of—it shares—it’s not just like a database. Life cycle. It’s sort of SDLC adjacent as well. And so I think thinking about the developer workflows that are required to build really great search systems and retrieval systems—you should think about it in that loop. And then, you know, obviously there’s, I think, a lot to do in enabling developers to do that. And so we’re still early, I would say, broadly in that arc, but we are working on a variety of features to help developers run that loop very quickly and efficiently and mm-hmm increasingly, AI [00:34:00] itself use that loop. So that’s one. And then there’s a little bit of, “skate to where the puck is going.” You know, you can’t always be purely reactive to your customers, you know, they’ll always ask for a faster horse, which is great. We also, you know, try to think about what is the future of AI search and mm-hmm, you know, what parameters need to be in place no one’s thinking about today, but everybody will be thinking about six months from now. And so we have our thesis there. But you know, those may be some—still some secrets.
[00:34:26] Victoria Melnikova: What is your stance on fundraising?
[00:34:28] Jeff Huber: I think you should fundraise when you have a good idea of how to go faster and you should fundraise probably more than you think you need. You don’t get points for shaving pennies. You get points for winning the big prize at the end of the rainbow. Yeah. And you shouldn’t raise funding if you’re not going after a big prize. VCs don’t care at all if you’re going for a small prize. I think the other way to think about funding is fundraising—at every stage of the company you’re trying to prove something. And you can’t prove everything at once, which is just, you know, natural. And so thinking about fundraising in terms of milestones—for this milestone, what are the things we need to prove? And then you prove those things, and once you prove those things, you both proved to the investor, but you’ve just proved to yourself that there’s something here and then that’s the natural point to go and raise more and go faster.
[00:35:13] Victoria Melnikova: I guess my final question would be, what do you think is still missing in AI tooling in general? Like what are some big gaps that you’re seeing? Just to give our listeners ideas on what areas to pursue.
[00:35:27] Jeff Huber: Yeah, it’s somewhat difficult. I think that one simplistic way to think about AI is, AI is ultimately just like reasoning plus search. And so, you know, the labs are working on perfect reasoning. We’re working on perfect search. I think that a lot of the stack will get hoovered up by kind of the reasoning side and the search side basically. And even the opportunities that look interesting today, you know, could become marginalized as a result. If I was a developer, I mean, yeah, if I was a developer, I probably would work more on the application side than the infra side or the tooling side. Again, tooling open source is really hard to commercialize a lot of the time, like really hard. I think there—I mean there’s a lot of interesting questions around what massive scale agent inference will look like. This means, you know, today we run like one agent then talks to one agent talks to one agent. Yeah. But in the future, it could be like one agent kicks off a job across 10,000 agents. And then another agent judges which out of those 10,000 was the best. This has actually been shown to be very effective in research and academia and there’s just a lot of pretty interesting compute orchestration problems. That—you know, obviously you have the GPUs themselves as a constraint. Yeah. All the bottlenecks therein. And so I think there’s an interesting opportunity for a new kind of distributed inference execution platform. And I think that that probably could be open source, but then, you know, running that system would also be fairly complex. And so then maybe owning that system, you know, offering a service around that system, you could monetize that. But, uh, it’s very hard to know, you know, it’s very hard to know.
[00:37:06] Victoria Melnikova: I’d like to end my conversations with one question, and I think here it’s most fitting because by the sound of it, you love what you do. Chroma sounds like a product of love very much. Mm. And I think users also second that. Right? So you get a lot of kind of love in response to that.
My final question is what I call a warm fuzzies question—what makes you feel great about what you’re doing today?
[00:37:33] Jeff Huber: I think the bull case in my mind for AI is, you know, yeah, maybe it’ll cure cancer and whatever. All this crazy stuff and I don’t know—it just seems like that still seems mostly a pipe dream. I believe in pipe dreams—pipe dreams are great to believe in, but it’s mostly pipe dream. I do think that AI can lead to the democratization of services though. It seems very clear that even the models at their current capabilities, this is entirely possible. And so you can offer world class education, medical services, financial services, legal services, et cetera, to anybody for pennies. And I think there’s—that is going to be extremely impactful and beneficial to all of humanity. And so, you know, maybe, you know, when you build a tool, you can’t always control how it will be used. You know, you build a hammer, right? Hammer’s been used for a lot of good stuff, all this bad stuff too, you know, I don’t know—read the news. And so, you know, you can’t always control how it’s used, obviously, but, you know, it always is exciting to me when I see, you know, customers using our technology to, you know, it’s cliche, but, you know, make the world a better place. That’s very motivating.
[00:38:43] Victoria Melnikova: Finally, I want to provide space for you to invite people to try Chroma. How can they find it? Where should they start?
[00:38:50] Jeff Huber: Totally. Yeah. If you type Chroma into Google, it’ll be the top result.
[00:38:55] Victoria Melnikova: Hopefully. No, I’m joking. Usually is. Usually is.
[00:38:57] Jeff Huber: Or trychroma.com and you can sign up. You get five bucks in free credits, no credit card. You can load in some sample data. I think that five bucks is equivalent to like a hundred thousand documents and a hundred thousand queries. So it’s actually quite a bit. And yeah, just experience, I think, how easy we make search and retrieval. It doesn’t have to be hard anymore, so.
[00:39:19] Victoria Melnikova: Awesome. It was a pleasure. I really hope that everybody else who is listening will also enjoy it—and watching because we’re in this amazing studio. Thank you for hosting me and thank you for coming to my podcast.
[00:39:32] Jeff Huber: Thanks for the time. This is great.
[00:39:35] Victoria Melnikova: Thank you for catching yet another episode of Dev Propulsion Labs. We at Evil Martians transform growth stage startups into unicorns, build developer tools, and create open source products. If your developer tool needs help with product design, development or SRE, visit evilmartians.com/devtools. See you in the next one!
