Automated transcript, aligned to the final episode. Timestamps open the corresponding moment on YouTube.
0:00 swyx · Intro — Okay, this is a pretty special episode because we're celebrating 1 million subscribers on the AI Engineer YouTube channel. So go over there and subscribe if you haven't heard about it. The guest is Mihail Eric, who is not only teaching The Modern Software Developer at Stanford, which is going viral right now, for basically covering all the AI-native development best practices, but also he is head of AI in Monaco. So he's not only a teacher, but he's a practitioner.
0:24 And those of you coming to AI Engineer CODE Summit next week, it's going to be the celebration of all the extreme progress that you've seen in coding in the last year. I'm excited about all the themes of fighting slop, of system one models, and of increasing your ambition for major rewrites of basically everything that you've seen in code. There's never been a more exciting time to be an AI engineer. And with one million AI engineers, I think it's just getting started.
0:49 So let's get into the interview.
0:51 Okay, we're in the studio with Mihail Eric, adjunct professor at Stanford and head of AI at Monaco. Welcome. Thank you. I'm glad to be here. You caught our attention because of your course that you are refreshing for the third time, third time?
1:03 Second time. Second time. Yep. The CS146, or 146S, The Modern Software Developer at Stanford.
1:09 Mihail Eric · EO Studio excerpt — This particular generation of junior developers, of junior engineers, the people that are now entering the workforce, will, I think, be the first kind of generation of that new shift.
1:18 And I looked at the syllabus and I was like, this is everything we cover. So every now and then I like to invite people like yourself who like you cover a broad range of topics and then we can sort of go topic by topic and Give people the state of the art on each of these things, which I assume you're doing with all your guests anyway For sure. Tell us the history of the course.
1:36 What made you want to Spring out CS 146? Yeah, is the second time it's being taught and I I taught it for the first time last fall, so fall 2025. This was pre 4.5, 4.6 days. even before fall 2025, the course had been about a year in the making. The genesis of the class was probably November of 2024. And I had sort of had this idea that we're seeing all this stuff happening in coding agents, and yes, they were an earlier version of the coding agents, but already it felt like there was something drastically changing in terms of how developers were doing their work, quickly we were getting work done, all that fun stuff.
2:08 And so at the time, was all just scattered blog posts, scattered things you could find online if you were trying to get into the methodology of how to do these things. And so I had this idea like, hey, well, is there a way to kind of structure a curriculum around this that's a little bit more formal?
2:23 Had the idea, and my alma mater is Stanford, and I always had this to go back and teach something and kind of pay it forward. I've always been very deep into education. And talked to Chip who also taught a class one of my good friends Chip Yen yeah Chip exactly spoke to her she had taught her I guess designing machine learning systems and so like asked her what that process was like and sort of you know took a lot of inspiration from how she designed the curriculum and thought about you know getting something the school put it together spent about nine to ten months this was now fall of 2024 through to like you know mid to late of 2025 got it through the committee,
2:58 made a case for the class to exist, you know, had to put together a proposal and everything. And that took me to fall 2025 when I heard it for the first time, was very well received. know, students were really interested in it. There was even, I guess, like a Business Insider article that was written about the class last year. And then, yeah, this year, a lot of things changed. I would say really agents, autonomous coding agents, really picked up in a way that we had not seen before.
3:23 So it felt like it was time to do this again, refresh things, and then here we are. Meanwhile, you're also head of AI at Monaco, which is also a very buzzy startup. Yes, yes. We are in the airwaves right now and having a fun time doing it. Yeah, so I do like that you're also, I guess, a practitioner. You guys are obviously implementing all this stuff at Monaco. Yeah. Any, I guess, principles or observations about just have changed from last year to this year at a high level?
3:53 Yeah, yeah. a ton. I think that when I reflect on what last year's version of the class was, I mean, coding agents had been around, but we were still in kind of that phase where companies were emerging that were tackling every single part of, like, the AIS DLC. There was, like, a coding agent for code review. There was a coding agent for spec design. There was a coding agent for SRE. And, you know, some of those, of course, some of those companies still exist.
4:16 But it felt like last year it was a lot more about a broad tour of here's everywhere you can use coding agents. And in a way, a lot of the curriculum last year was geared toward exposure. And this kind of survey, this kind of like odyssey of sorts through, hey, everything that you knew ASDLC is now changing because of AI. This year, I think there's some principles that are solidifying, right?
4:38 The idea of like very careful context engineering, the idea of like really thinking through sort of like design and architecture as a way of handling really complex tasks. And so the nature of my class is it's not intended for the like beginner programmer. That is not a replacement for like traditional computer science in the sense of like algorithms. And if in fact a lot of those classes are prereqs to what I'm doing here.
4:59 But I do know that you have a good foundation, going now toward applying that and sort of using those principles in production environments in the real world, that's something that I think is now at a place where, you know, at Monarcha we deal with massive code bases. You know, I help a lot of companies also with thinking through their massive code base designs and things like that. And so it's more of a practitioner's course for once the students, graduated or now entering the workforce, how to be the most productive members workforce.
5:25 You said you're passionate about education. I feel like students in that phase of being in college are in a pretty interesting spot. any advice and any take should people study computer science right now?
5:40 Is Claude code and codex all we need? Like, I want to, you know, hear your thoughts. Yeah, I, you know, there's a lot of anxiety from students, as you can probably imagine. And I think a lot of what I'm trying to communicate this time around is that the anxiety, a lot of it still is here. And there was anxiety last year as well. But I want to make the case that a well -trained AI native engineer, and I know that's like a buzzy word, but let's say we'll dive into maybe what aspects that are most interesting.
6:10 A well -trained and native engineer can be some of the most useful assets to any company around.
6:17 This is the first week of class, actually. Week one, first lecture, I'm like, hey, I really believe that every company needs you, but the floor is much higher. You're entering as your first job out of school, but it's not sufficient to just be an L3, in a Google term, if you're going to use that kind of leveling system there. You have to generally be better than almost at the senior level going to this.
6:39 So like what can we, how do we get there more quickly? Right? And so I'm trying to say that there is still a core bit of like foundational software engineering that is important to have. Some of these core values of sort of how software systems are built and the trade -offs of those systems. then beyond that we go and, know, that's where I think you as the humans should still be valuable rather than, and the rest of the implementation details you can kind of defer more often.
7:02 Hmm. Yeah. This is something I've said before which is almost like the controversy it's like the kind of thing you say to be spicy but I almost I do genuinely kind of believe it which is most students are cooked some of them are going to be super valuable I don't think it's a very popular thing Stanford's kids are going to be fine right they're from Stanford but like what happens to like the rest of America right that is an over question it's a weird situation because I feel like a lot of education is slow to move like I'm even curious in how do you determine where do you put the end date to curriculum in a course on it?
7:39 You're teaching a Stanford modern software. That's changing everything. You cheat. You just get the people from the coding agents to come and speak, right?
7:48 Oh, yeah. Have them be the, yeah. That's a funny. New old guest lectures. Well, yeah. I mean, that was, okay. So on the guest lectures point, I think that was, I mean, that was also last year. That was also the case. So that was by design because I, look, I came in, right, and I'm saying, I'm a practitioner, as you pointed out, swyx. Like, I'm a practitioner. I do this thing on my, you know, my day -to -day.
8:09 I design these systems. You know, build these things in Monaco have seen that many times before. And so that being said, don't want to claim that I have all the answers today. And that would be, like, very, you know, a lot of hubris to make that claim. So the structure of the class was always here's one practitioner's sort of aggregated set of insights from working in the field, from talking to a lot of people in the field.
8:33 But then there's people that are like literally on the frontier, you know, seeing things even beyond that and invite them to sort of put their proposals forward, their view of the world forward. And that's why the guest lecture is there. I think it's really impactful for students to see big names, people actually doing it, like hear from them directly, be able to ask them. Of course, you know, they want to intern, they want to get jobs.
8:56 It's good to have guest lecturers. No, I think what he's saying. So, OK, I'll sort of rephrase. No. So, OK, there, you know, we've also interviewed Ange, who teaches the Coachella course. Right. Right. Which is like, here's all the names. It's only the names. Right. Whereas like he's actually also producing, you know, here's cloud code in 200 lines of code. Right?
9:15 Which is, no one's telling you to do that. No one's gonna come speak to you to do that. But like, yeah, when you wanna teach fundamentals and a framework that you can use and working understanding, this is the kind of thing you do. That's right, that's right, yeah, yeah, exactly. And I do think that there's a balance for this kind of material, to your point, is that there is a, you can't, it's not 100 % solidified.
9:34 I mean, by design, it's a feature and maybe a bug of some kind, but a big portion of the curriculum did change from last year, right?
9:42 And so there's like a dynamism to the whole development of the curriculum. But I'm trying to at least put forward that there are some things that I believe are kind of sticking around. And yes, prompting coding agents, they've gotten smarter. What does that mean in terms of the new ways of dealing with them?
9:53 But then still, Boris is also going to know something that I don't know, right? So let's have him also come to the table and share his learnings. One of the fun things I noticed this time around is a good bit of the portion is on open source contributions. What's the thinking behind that?
10:11 You have people… Yeah, yeah. This is one of the things that I’m honestly kind of most proud of and most excited about. I mean, this is about to be week two of the class. Yeah, so this is the very gracious open-source partners that I spent the greater part of this summer just kind of talking to
10:28 and getting them kind of excited about and interested in sort of this offering. Basically the idea was, so last year when I taught the class,
10:34 we had like nine assignments, and then we had like a capstone open-ended project, right? The nine assignments were basically me sort of like coming up with these pedagogical exercises. Like, oh, here’s a repo. Use an AI code review tool to, you know, post a code review, and then see: why is it good? Why is it not good? So, relatively pedagogical, honestly pretty easy. I think for a lot of folks, they could just like one-shot it, and I thought, okay, this is not that helpful. At this point, agents are around. I mean, they have—every student that I surveyed, you know, on Tuesday, not a single person really types by hand anymore. They’re like, everyone’s using coding agents. And this is, you know, like all sophomores, juniors at Stanford.
11:10 Do you survey them? What's the market share? Was that like what, what tools they use? Okay. I haven't yet, that's a good call. You should do it. Yeah. Yeah. I will do that. Spicy, Stanford prefers. No, no. Well, I think university, I think they'd have some, like most of the providers have like an educational program, right?
11:27 I mean, yeah, they're very generous on plans for students. Right, right. So I think by default, even before the class, there's like, you know, Claude license, I imagine cursor license. There's like probably a cognition license as well. don't know, you might know that better than I do. But I was able to also just accrue on top of that was, hey, like if I don't want people to be bottlenecked by the $20 a month subscription.
11:49 I want you guys to go, if you want to token max something just to see what's possible, then you should. And so, I'll come back to the partners, but the additional thing I was able to accrue through instead of talking to folks was additional credits for students. So like factory is also getting involved, the cognition folks are getting involved. OpenAI as well. But anyway, so for the open source partners, I sort of was like, okay, one of the things I'm most concerned about when it comes to educating of students is that if everyone is saying, kind of to Suix's point, like they're cooked, like what's going to happen to them?
12:19 There's almost like this sense of them being left behind. And I think that mentorship is a really important piece of just up -leveling everyone and making sure that there isn't this massive gap that occurs between the senior people that have already been in the industry before coding agents and know what's up and they adopt coding agents and they become amazing. And then the juniors that are just like, okay, now what do we do?
12:39 So in a sense, one of the kind of goals was, hey, coding agents are really good. I want to put you guys in situations where you're dealing with complex code bases, large code bases at scale. We're going to work with someone like what I think are among the best like AI native repos, you know, that all do AI focused things or either coding agents themselves or building tools in the service of agents, you know, in the modern world.
13:00 I want you guys to get involved with these projects. I want part of your assignments to be actually contributing to these projects. In the course of being in all those projects, you will actually be talking to the maintainers. They will be reviewing your code. They will be helping you build mental models of good software. In that way, there's effectively this passing on of the guard of true good software design in a way that I don't think most students get the opportunity to do.
13:20 And then there's the benefit, of course, that I'm sure some of them will get poached and picked up as part of doing well in those assignments. Yeah. One of the interesting things, for what it's worth, get a bunch of hate if I I don't say this. I think you say cooked versus cracked for dramatic effect. know, like, it's just nice to state extremes. Sure, sure. Yeah. So maybe we should go into the sort of detail of the curriculum.
13:46 Yeah. Is there anything else you want to explore at the top level? I mean, like, that's very helpful. Yeah. All that's there. Okay. I have high -level questions about students using tools, best practices. Are you pro it?
13:59 Like individual tutors. I'm assuming this will also be picked up by students. So, you know, before we dive straight into the course, just anything you have guidance on there for. Like in terms of what. You said you're passionate about education, right?
14:13 Like, yeah, there's also a stigma of side is how do teachers, professors use it? And then how can students best use these tools? That's not just throw your assignments in. It can kind of one shot it. How do you still learn if you have anything there?
14:28 Yeah, yeah. And I think that there's, still have some assignments thrown in there, which I think are not intended to be the hardest things in the world, but are meant to be more about concepts, like more conceptual things. Like, you know, what does a skill mean?
14:39 Sort of like maybe more of the tactical of some of these tools, skills, like agent code bases, like agents MDs, things like that. So there's a certain amount of just, hey, if you haven't heard these terms, kind of get up to speed on them. I do think most of the learning, though, in terms of the of putting the tools to the test will happen both in the OSS and then also final project where it's very open -ended.
15:00 And they'll have to just kind of explore something interesting. Cool. Yeah. So maybe, yeah, I just want to talk about, well, let's go through one, two, three, four, uh, to 10. Yeah. Uh, and then we'll dive in deeper. So like breath first, basically. Sure. Sure. Yeah. So we start with internals of coding agents. That's right. You teach people how to write their own Claude Code clone. It's very helpful that it's not open source.
15:23 And then? Yeah. Yeah. So, uh, week one, it's all, uh, internals of harnesses, you know, kind of build a coding agent from scratch. Uh, week two is more, okay, now assume that there's, how do we start thinking about like, context engineering at scale. So beyond just vanilla prompting. And then I think coupled with that is just, in doing that, thinking about tool design, MCP. People, I think for a while MCP was getting a lot of hate.
15:46 I'm still pro -MCP, I see it all the time. And so I think that there's, at this point, it's a standard that's existed and I think it will continue to exist. So talking about that, effective tool design, that's week two. Week three, I would say is more about just like skills and what does it mean to extend beyond simple purpose tools to more like procedures. And that's where we'll talk about skills.
16:06 And that kind of starts leading into the automation of agents. Week four is a lot of just of like the foundational pieces of how can you customize an agent. This is things like config files, a la AgentsMD, CloudMD, hooks, sub -agents. There's a lot of mechanisms that exist in modern harnesses which you can use for different purposes. So I want to expose students to that. Agent -ready code bases. This is probably one of my favorite topics.
16:33 It's like, and this is especially relevant for folks that are going into industry and thinking about at scale is, you know, essentially what does it mean to have a repo that's ready to be consumed effectively by an agent?
16:45 And so thinking a lot about, you know, the trade -offs of designing of of service level boundaries, but also docs and verifiability and things of that nature. So that's week five. Yeah, week six, sort of agentic code review. And here I think what is interesting is thinking also kind of extending from week five, week five around code base, like agent -ready code bases. Going to week six is more about like, okay, in order to, like code review is now becoming a bottleneck, which I don't know that students, unless they've, you know, have done a lot of, some of them are master students, so they've been in the industry, but a lot of them have not seen this.
17:17 But for most people that are using code agents today, we're all experiencing something of a new bottleneck in code review. So how do we unblock that or sort of unbottle the bottleneck a little bit?
17:27 What mechanisms are there there? And so verifiability and things of that nature and how we can extend that into the code review process. Week seven, security. I mean, obviously, I think there was a point. So this was in last year's class as well, and this I wanted to reinforce. I mean, with any autonomous system taking a lot of actions, we're seeing a lot of this year, especially a lot of things getting hacked.
17:48 You know, not least of which are just some of the core pieces of software from Vercel to, know, gateways, LM gateways entirely. So I want to spend some time just like talking about what new attack vectors exist in security and, you know, how we can mitigate some of them. Week eight is background agents. So this is what I want to start touching on here is, you know, how do you like basically run agents in many different product service areas, not just in what was traditionally an IDE, but also having things that are based on triggers in your code base and in your repo during CICD, things that get triggered even when based on events, things like SRE style bots.
18:31 How do you even build a system like, I think this past year was when a lot of companies started releasing their posts like ramp inspect. Spotify has their own version of this, I think Stripe Minions, so all these folks were doing really cool write -ups of their internal infra they were building to do this. I want to touch on that, get people exposed to what that is. Yeah, this one is more on what happens when you're in a team the modern world and sort of like the core pieces of team -level infrastructure.
19:00 So everything from LM Gateways to MCP Portals, which is like MCP at scale, and then just also how to think about the evolving roles, I would say. So this is maybe a little bit like a soft thing. It's like what is now, you know, and the vanilla, maybe like not super nuanced thing to say is like everyone's a builder, but I want to go a little bit deeper beyond that and like sort of like what thing is actually getting subsumed?
19:22 What roles are actually either evolving or emerging that didn't once exist? What should students think about in terms of defensible skill set that's maybe wasn't true before?
19:31 And so a lot more like team dynamic type considerations. And then, yeah, this is it all kind of builds up toward the factory, which I think is top of mind for now. for a lot of folks today. Once we're building factories, do humans still exist and all that?
19:46 What are the new choke points that are intentionally there around human involvement? Things like intent, things like setting up a verifiable systems. And what happens in the limit when maybe 80, 90 % of code is fully driven by just agents, where we kind of filled into all that. So, yeah. Nice, solid 10 weeks. I was gonna say, like, know, some of these are more established patterns than others. And, I think people need to know that, like, not all of this is settled science.
20:18 Sure. But it is everything, I'll endorse everything as things that people talk about and you should know. We know the open problems, we may not know the answers. That's absolutely right. Yeah. I mean, you probably have your own version of this. I'd be curious to hear what you think is still not as defensible, or not defensible, but not as set in stone. But even like the software factory, are we talking about dark factories?
20:39 Are we talking about these sort of half lights on factories? I think even there's a spectrum there. Yeah, and there, you know, the people that are on the very frontier, like the warps and the factories that are like building factories and selling them services. And, you know, even they only claim like 30 some percent of even that of like fully, you know, agentically driven So, what does that mean in the limit?
21:00 It is hard to say. And there's people who went very, very far dark factory and then came out saying I was wrong, guys. Yeah. Right. So, like, we covered that on AIE. just be aware that not everyone, they probably believe the things they're saying at the time but they have the right to change their mind and they do and even Boris recently tweeted that yeah I am often wrong and everyone is like, because he created Clock Code, it's like treating his word as gospel and I'm like no, sometimes he disagrees with himself and that's okay this isn't politics, right we're still scientists, we're engineers we're subject to, I don't I don't claim to be dogmatic, but I just want people to experiment.
21:40 I think that's a principle as well. You asked about principles earlier. Just experimentation. I mean, you just have to be ready to ... And oftentimes, when I give these lectures or talk to people about this stuff, they're like, there's so much going on. How do I stay on top of this?
21:52 There's something as simple as just experimenting for 10 minutes a day. Just spending 10 minutes doing something that you didn't do before. If that's a traditional way that you were solving a problem, there's now a new tool that you want to try or a new framework or something. And just spending, maybe even doing the exact same problem just with that new kind of thing alone will cover most of the gap. Yeah.
22:10 I will call out, so some of your, for example, you have Cognition on the week that you're talking about Code Review. Yeah. I actually did work with Silas on Devon Review. Yeah, yeah. Pretty good, I would say. hard to benchmark these things. There is a ReviewBench. No one uses it. But qualitatively, Devon Review is pretty good. Well, we use you guys. Oh, really?
22:30 Yeah, so Monaco does use Devon. And Silas actually spoke last year, and so Silas is great. It's just super, super smart. But yeah, we use you guys. And so I was like, okay, among the providers of kind of code review tools that we, and we sampled a lot of them. We actually did kind of converge on Devon for a lot of use cases. Silas is actually of a model side these days, and then actually Walden that you want.
22:51 Yeah, yeah, yeah. I spoke to Walden as well. I will also shout out, you know, from Factory, because, you know, talking about agent readiness, They've been all about that. They build signals, tooling on that. Do you guys use that?
23:02 Yeah, yes. So we do use agent readiness of our own kind. I mean, Eno is awesome as well. I think they were super early on this train. think when he codified that in his talk, I guess, at AIE, what was it, like last, I forget when it was, but one of his talks. Europe, yeah. Yeah, Europe, just fantastic talk. And I think it was kind of way ahead of its time. And so I was like, there's only one person that I could see being the most appropriate person to talk about this where I really want to give them the spotlight and super excited.
23:27 Yeah, it was funny. I was at Cog. We were talking about this stuff and we were like, okay, we need to form some opinions about it. And then like Factory like shipped a full product on it that week that we were talking about it. I was like, okay, you won. I mean, like everyone kind of agrees on the same thing. And it's just a question of like, do you want to turn it into a dashboard?
23:45 Because it's a dashboard. Right, right, right, right, right, right. But you know, there's value. And like, I think they did well from a marketing perspective jumping on the writings and report. And I've definitely spoken to, you know, I will regularly kind of pitch that to other people and say, like, yeah, you should run writings and reports of your own kind. And, you know, of course people have open source versions of it, but I think the principles are super important.
24:04 Yeah. Do you want to go piece by piece or do you have any other things? I can go piece by piece. Yeah, okay. So the first one, right?
24:11 You focus on cloud code, but then you also have Pi as an open source sponsor. What's up? Okay, so this one is maybe a little bit more marketing -y, but okay. The one thing I do want to highlight, and I've made that, I also made that clear last year, but I'm not endorsing any tool, okay?
24:26 Like, I don't, I think that is the wrong way to approach something like this. There are tools that come and go, and you know, transparently, when the class was taught last year, Cloud Code was not as popular. It wasn't as, like, really accepted. Boris was only at 4 ,000 Twitter followers, and now he's at, you know, whatever million. I don't even know if he is. He's had a lot of followers. Cloud Code has really, like, emerged on the scene.
24:49 I think what. Yeah, I think we, we also underestimated Claude Code when we had Boris and Kat on the podcast. Right, right. It was, like, two months old. I hadn't really, like, I used it once. A query cost me 50 cents. I was like, who's going to use this shit?
25:00 Totally. It was still in beta for a while. But I think it's good on them for how they built it before the model was ready. Yes. Like, you hear so much build for where the model will go. Totally. Super ambitious. Like, models were not that good. You couldn't feel it. And then, ah, we see, we see that. That's where it came from. Yeah, it's funny, because when he gave his talk last year, I remember, it was like a very high level of set of slides, but I remember one of the slides was literally built for where the model's gonna be six months from now.
25:23 And this was like October of last year, and I can see why he would've made that point if he knew in six months where Opus was gonna be, because that was also what I think was a huge unlock, where I would argue is one of the reasons why the curriculum changed as much as it did, right?
25:37 Where there was certainly this huge arms race around true autonomy in these agents, like long -running agentic agent tasks. But yeah, so back to your point, it's like less on, I'm not endorsing Cloud Code. I mean, Cloud Code, of course I use it and use other things as well, but here I was more about like the, what does a modern coding harness look like?
25:54 I wrote a blog post actually, yeah, this one, this was earlier this year, where I sort of broke down some similar -ish principles The TLDR was that you could sort of, through like pretty much introductory algorithms level understanding, could just find like essentially an agentic loop that does the bulk of what you would call a coding agent. And of course there's like bells and whistles, there's no sub agents, there's an MCP and none of that stuff.
26:18 But if you want to see something that you talk to and then it literally will edit your file and you can do pretty sophisticated things with it, it doesn't require more than 200 lines. So a lot of what I try to do in education is to try to peel away the stuff that's like the secret sauce show that really at its core, I mean, these are still students and you want to teach conceptual principles.
26:37 There's nothing more to this than, in Stanford, there's this class called 106B. It's like the introductory algorithms. You don't need more than 106B to build something like this. So you go. There it is. Given that you already taught this class, pushback, any questions, any weirdness that you observe from the students?
26:54 What do they not get? What do they seem to cluster questions around? You mean like now that this particular concept or just in general?
27:02 This concept. Of like building it from scratch and then doing what you did. Because this is basically what you needed for Pi. Right, right, right. Exactly right, yeah, yeah, yeah. So it's not really. I think they're, I haven't received any pushback on this idea at all. I think they're actually more just, most of them are consumers, right?
27:21 Most of the students do just end up using the tools. I don't think that they get a lot of exposure, if any, certainly in the coursework, around peeling that onion back. And I think, so this was the week of the harness. So Tuesday, first lecture, logistics, blah, blah, blah, all that stuff. But then first half is, the first true bit of curriculum is the 200 lines of a harness. And then yesterday, when I had sort of the follow -up to that, is we actually took Cloud Code, and then you have these proxy things, You need like for proxy tools where you can intercept the request.
27:53 MIT and proxy? Yes, exactly. Exactly right. So use that. Use that tool. Intercept the requests and responses. Nice. Look under the hood, see what's happening there. And then just like, OK, I did say you have Claude code in 200 lines. But really, OK, here's like actual Claude code. And so what I literally did is I took Opus 5.5, so this model of two days old, and was like, this is, I intercepted those requests.
28:13 Here's exactly what's going on. Here's the prompt. Here's the system prompt. Here's the plan prompt. Here's the compaction prompt. And then we just went through all of that and sort of showed it to the students. So now they really, really have like a state of the art harness that they can understand. Yeah. It's nice to get those three levels. Like students are very much consumers. Here's the baseline agent in a loop and then the real one.
28:32 I like how you have a video walkthrough of your blog post as well. Do you see students are on like extremes of some people are power users. They consume a lot. For some people, you could assume that this is kind of very straightforward if you know how agents work. How do you kind of balance that in the classroom setting?
28:53 Like do you tailor towards the average? Yeah, I think it's a bit self -selecting because the curriculum is all here, right? So the curriculum has been available since, I mean, you know, left and least like a few weeks and I guess I announced it a few weeks ago but it had kind of been in the works for like probably at least one to two months. So if you looked for it, you could probably find it.
29:12 And I feel like reality is if students are not interested in this, then they won't like, they just won't show up and they will choose not to show up. What I'm finding that is interesting is there are definitely students that show up, but then there's also just a lot of folks from broader Stanford, postdocs at other departments. There's a lot of like interdisciplinary sort of interests where folks that are in, you know, somewhere in pathology or like, you know, like medicinal sort of fields or, or even like random other engineering fields that aren't pure CS that are coming in and also interested in, in kind of getting exposed to this.
29:41 Well, also a lot of, uh, GSB students, MBAs, um, that also, uh, want to kind of learn how to not have a technical co -founder or whatever the motivation there is. But so I think, by the way, if anyone hasn't done the MITM proxy exercise, you should do it. Yes. You'll be shocked. Cloud Code has the longest system prompt of all the coding agents. Something like 20 ,000 tokens. Yeah. Yeah. Although, yeah, that is true.
30:07 That is true. Although they've also reduced the size. They cut it by 80%. Yeah. Yeah, I know. But then they grow again. And then they cut and grow again. It's a seesaw. Yeah. Yeah. No, I think it's interesting. You'll be surprised what goes in there. That actually sometimes you don't use. Sometimes it actually hints at the internal workings of some of the tools. Anything that you found there that could be cool to talk about?
30:28 Yeah, no, I mean, in fact, yesterday when I was talking about this, it was a little bit of this like, one of the things that I thought was most interesting about Cloud Code back in the day, and I think part of why it really blew up in terms of popularity was their focus on simplicity, right?
30:41 So in the early days, they're among the first to really lean into this, lets fewer tools do more, right? Lean into SQL, lean into Bash, lean into the things that are powerful, like Swiss Army knife tools, and have that be a lot of the driver of capability. I think they did a fantastic job leaning to that. I think that's why they were as popular as they did. I think that when I presented something like the system prompt, it was like, well, there was a different system prompt about two months ago.
31:05 It is now, they have reduced in complexity, and it turns out that now what was once part of the system prompt, things like how to use even Git, right?
31:13 or how to set up tasks or things of that nature are no longer part of the system prompt. It's actually expected to be more captured in the models, like the model weights itself. So they've kind of baked in a lot of those learnings more at the model level. And that just raises interesting questions around the model versus the harness, right?
31:26 And how do you kind of think through that trade -off? What do companies invest in? What should you invest in to really push the frontier of your own capabilities?
31:34 And so I think we still, even as a community, we see saw between the two a lot. I think you don't want to fault them for growing a system prompt. And then cutting it. At some level, it's also how the model was, right?
31:47 Weaker, older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right?
31:54 And that's just part of learning what has been done, right? You can apply the same towards if you're using a local coding agent with not the strongest model, you might need to add these examples, right?
32:05 That's right. Is that correct? Yeah. Yeah. And then, of course, there's interesting AI learnings to follow out of that, which is why, you know, as the models get better, when are you over constraining it by adding those examples, right?
32:16 And then how do you pull away the over constraining? And that's, I think, part of the learning as well. Okay. Week two, you're going to talk about context engineering. Repit, I haven't seen, but there's many versions of Spectre of a Development. What would you, would you highlight to a broader audience about what you're putting in here that people should know?
32:35 Yeah. This is like the, you know, one hour version of the course. Yeah, exactly. Please condense everything. I think that's where this is going. That's fair, yeah. Well, no, it's also a trailer, right?
32:47 It's like, well, if you want more, go to the source. If signing you're up next year or next fall for courses, you know, this is your one hour. This is like, this is like, oh, it was like the Quibi of the class. You know what I mean?
32:56 It's too long to be a trailer. It's too short to be the class. Is this like, what I guess people will double click. If it's a trailer, by the next time you teach it, it will change again. That's true. That's true. That's true. That's true. Yeah. Well, yeah. So it's okay. So good call on that. It's funny. I think that the main thing that I just want to reinforce here is, and I see this even when I'm like interviewing folks, very few people in industry even progress beyond like just vanilla prompting, right?
33:21 There's like a form of like, hey, do me the thing, right? Like do the thing. And that certainly as the models get good, that can carry you a certain distance. But inevitably, any folks that I speak to that come out of a Google or someplace where there's a massive code base, the reality is you have to be a lot more thoughtful about how you're ingesting context. So thinking through what are some of those principles and those techniques around how to point ...
33:46 The rapid framework that I'm describing here is functionally similar to like DEX's RPI, right, or something of that nature, where you have this like research, kind of proposal, sort of a spec, and then you do like your implementation, and then you kind of tie it together with some testing and whatnot. So what I just wanted to reinforce here is that like you should have this step, when you're entering unfamiliar code bases, when you're trying to sort of infuse into the context something that an agent needs to understand about a very large code base, there's value in something that is a research phase.
34:16 And then in the case of this particular sort of framework that I'm putting forward, it's like also then suggest some proposal where there's a few alternative ways of solving a problem once you've done the research. And so this is where I sort of also invite the human in the loop to sort of be the, decision maker on one approach versus the other. And then you enter the plan, which is kind of the spec.
34:35 All to say that the, the TLDR is like, there is a way to do this that actually does scale out to code bases that are, you know, millions of lines long. Um, and it's not just the vanilla prompt. Uh, and I think that's what I want to highlight. if you're going to go working these big code bases, you should use something a little bit more involved. Anything to add on context engineering?
34:54 I think that probably the weirdest thing about how I think about context right now is when you split out to a subagent because subagents can benefit from inheriting your entire context of the main thread or not. Sometimes there's a regular disagreement on that. And then sometimes you want to condense what the sub -agent does and researchers put it back onto the main thread. And it's very hacky. I don't like anything I'm doing.
35:24 So I don't have any answers. All I know is when you want to fan out work, it is not just about you have a big spec and you RPI, you do Repit, whatever. It is not one agent anymore. It is about fanning out work and making sure they all have the right context. And they all have the right guidance. If you have a review agent reviewing all the other works, you have to make that guy have everything.
35:48 And you always go through this phase of expanding contracts. Like, oh my God, these guys don't have enough context. I'm going to give them more context. Okay, great. And then they're going to over to you into some random one -off comment that you made a while ago and then make some really stupid mistakes because you didn't mean it that way. But like, they took it as the law. Something like that. So I don't know what the word is I'm saying.
36:09 It's just like multi -agent management for context. Yeah, yeah, yeah. Yeah, I think that's absolutely right. There's like some aspect of this is maybe the part that's more maybe among the most art than science, right?
36:21 It's like some combination, I think, to your point. Yes, the fan out for the exploration, the fan out for like the subtasks, and then the compaction, whatever you might want to call it. I think being thoughtful about refreshing of the context, understanding when things are getting a little bit too bloated. I think Dex had this concept of like the dumb zone and sort of like being a little bit more thoughtful about the dumb zone.
36:39 He's good at these equating things. Oh, yeah, yeah. The little diagrams. Yeah. Oh, yeah. Why do you have him on? Yeah. Well, I think he's, I mean, he helped definitely informed a lot of the curriculum here. And so he's a fantastic thinker in all this. No, I think the diagrams are huge. That's useful. I would add breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences, like on your sub -agent inheriting.
37:02 Cloud managed agents has a version where sessions are separate and it's not inherited. This used to be a big topic before reasoning models when agents didn't run as long we had shorter context windows everyone was very very managing that precious context window right link chain used to write a lot of blogs about this in good context engineering all this stuff is good for people to know i think in the student context as well you can expect the average student to take your class because it's a great class and they'll know all these things.
37:35 So you should know them, know them better as well. This is a fun space for research as well. One of the papers, research papers presented at AIE, they looked at the delta in the cloud code system prompt, what got changed. And they plotted a whole bunch of changes in performance. They tried to reverse engineer it. It's just good research that you can do. There was another one that also looked at context window management and how it affects performance downstream.
38:03 They weren't the best papers, but it's quite accessible and it puts out good blog posts. So people that are trying to work on it, I just recommend it. It's a good area you can easily poke into. Yeah, good shout out.
38:16 Agent skills, CLI. Yeah, yeah. I think all this is a little bit of an extension from a single kind of function tool calling, like MCP, interacting with third party external services, and then extending into what you would call procedures. How do you automate some set of actions over and over again that you then formalize in the form of a skill MD or something like that?
38:38 think just making that a little bit more, giving people exercise in codifying some of those procedures. I think what
The original belongs to Latent Space and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.
Studio