The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.Today, we cover:Moving my video podcast off Spotify due to constant reliability issues. Spotify’s podcast platform has become chronically unreliable since the company’s leadership started boasting about high AI adoption. But competitors haven’t had similar issues, and so I have offboarded from Spotify.Will Kimi K3 trigger US push for close
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.
Today, we cover:
Moving my video podcast off Spotify due to constant reliability issues. Spotify’s podcast platform has become chronically unreliable since the company’s leadership started boasting about high AI adoption. But competitors haven’t had similar issues, and so I have offboarded from Spotify.
Will Kimi K3 trigger US push for closed-source AI models? Moonshot AI’s latest open model, Kimi K3, is on par with Anthropic’s Fable 5. Could it lead to the US government regulating or banning Chinese open models to protect US labs?
AWS laughs off “heart attack” billing error. AWS customers were billed trillions more than they should have been, due to what was likely a conversion error. But instead of sharing an incident report, AWS saw the funny side.
Industry pulse. OpenAI’s unreleased model tried to hack HuggingFace to improve its test scores, X took more than a year to develop its new Android app, Google’s new AI model flops, and more.
1. Moving my video podcast off Spotify due to constant reliability issues
Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. If you’ve been forwarded this email, you can subscribe here.Subscribe nowAfter I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick wit
Hi, this is Gergely with the monthly, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. If you’ve been forwarded this email, you can subscribe here.
After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.
This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:
Algorithmic programming speedrun. While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.
Eight years of infra at Shopify. There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.
“Napkin math” as a superpower. Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.
Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they were far more expensive than necessary.
A new product without VC funding & Cursor as customer no.1. After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.
Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.
Disclaimer: turbopuffer is a season sponsor of the podcast, but as with all our deepdives, this article is independent of podcast sponsorships.
1. Algorithmic programming speedrun
A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.
While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.
After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.
Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm, and dynamic programming; one of the more tricky concepts to master for algorithmic programming.
I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls:
“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.
Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”
Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.
Aware of how little he knew about programming, Simon doubled down to catch up. He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:
Simon’s desk with Donald Knuth’s “The Art of Computer Programming,” and the training week he created
As Simon recalled:
“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”
The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:
“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”
But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.
Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:
A five-line script to take screenshots and upload them to Imgur, all from the command line (2010)
Why Simon favors SQL over ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)
Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings. The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.
2. Eight years of infra at Shopify
With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke, which went viral on Hacker News, making it to the front page of the site with many comments.
It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.
In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. As he told me:
“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.
In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.
For example, if someone at work mentioned TCP, I assumed that surely they would know exactly what’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”
Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:
Ruby on Rails and databases: Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layer
Sharding and cutting over: as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.
Multi-data centers: expanding database footprint from one data center (DC) to multiple DCs
Splitting up key services: Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.
Simon built a framework to simulate networking conditions called toxiproxy. The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:
“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”
This was the inspiration that led to toxiproxy. As Simon told me on stage:
“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.
Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”
Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!
The biggest benefit of a long tenure at a company was learning to write software that ages well. Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.
3. “Napkin math” as a superpower
While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?
He sought the actual numbers, wrote a script to collect them, and created a table with the data:
Simon started to memorize key numbers with flash cards. He wanted to be able to instantly recall all important numbers. As he told me:
“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.
For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.
This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”
Napkin math enabled him to challenge design decisions based on benchmarks that had issues. As he told me:
“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.
For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.
So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.
Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”
Simon went even deeper into “napkin math” after leaving Shopify. He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:
“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!
Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could this system perform in ideal circumstances?
Rarely is the system 5x faster than the napkin math says. When we identify a significant-looking gap between the real-life performance and the expected performance, I call it the “first-principle gap.” This is where curiosity sets in. It typically means there’s (1) an opportunity to improve the system, or (2) a flaw in our model of the system. In this case, only (2) makes sense, because the system is faster than we predicted.”
Simon started to investigate, and learned that MySQL does grouping of transactions (not doing an fsync for every write), and it also does smart merging of multiple fsync operations that would be processed in parallel, effectively doing a “group commit” to further improve performance. This is a learning he’d later use: grouping writes together on top of S3, to reduce latency and improve cost.
Personally, I find it amazing how much you can learn with some measurements and by asking questions about how a system achieves results that go against what the “napkin math” suggested should be its limit!
4. Origins of turbopuffer
“Napkin math” also played a part in the creation of turbopuffer. Simon told me that three things had to combine for the project to come to life.
1. Search was a difficult project at Shopify. Building search was one of Simon’s final projects at the company, and he did not have a good time with it. He used a popular database vendor but struggled to have it perform napkin math. The query plans were not exposed easily by the database, so he could not figure out what was missing and why it wasn’t performing the napkin math without reading massive amounts of source code. Also, the search infra was difficult to operate.
2. Napkin math became a surprisingly efficient reasoning tool. As a toolkit, napkin math gave Simon a way to reason about what could be achievable with a machine, if utilized perfectly.
3. His friends’ startups needed fast search badly because of AI. In early 2023, Simon helped a few friends with infra at their startups, many of whom had the same problem: LLM context windows were very small (4-8 KB), and they needed to fill them with the right parts of documents, but this demanded very fast search. One startup budgeted that their new search vendor would cost $30K/month, while the existing infra bill was only $5K/month! For a bootstrapped Canadian company, the cost was too high, so they didn’t ship the AI feature.
Simon could not stop thinking about why search was so expensive, and why it didn’t line up with what the “napkin math” predicted the cost should be. Eventually, he laid out a basic architecture to make it fast and cheap:
Store the data to search in AWS S3
Do some clustering and organize the files
Get latency down. S3 is cheap and reliable, but has high latency
Driven by curiosity about how that could work, he set out to build it. As he told me:
“I just became fully obsessed that summer [of 2023] with building it. The first version was the simplest possible thing. I’m a very pragmatic person, so I didn’t get buried in detail. I barely read the literature on log-structured merge (LSM).
The simplest way to do this is to run a clustering algorithm on the vectors. You get the clusters, and then put the clusters in files. The files are called ‘cluster_01’, ‘cluster_02’, ‘cluster_03.’ Then you have a file called centroids of the clusters. Then, you search by downloading centroids, looking into them, then downloading the closest clusters.
There were a few optimizations around merging clusters that were adjacent in files, just to control costs and boost performance. But this was the core of it.”
What about performance? Simon:
“For the first version, I didn’t even implement a dedicated caching layer. I just put the reverse proxy (NGINX) in front of S3, and that was it! The performance improvement came from caching all of the S3 objects. It was the simplest possible layer at the NGINX level.”
5. A new product without VC funding & Cursor as customer no.1
I first heard of turbopuffer last year, when I interviewed Cursor cofounder Sualeh Asif about how they built the AI coding harness. From our previously published Cursor deepdive:
“The ongoing need to re-shard as usage grew became error-prone and frustrating. Sualeh told me the biggest lesson the team learned was to avoid sharding where possible in future. So, they looked for a vector database that could support multi-tenancy without requiring manual sharding.
Turbopuffer was a startup that promised this, so Cursor tried it and migrated a good part of their vector search use cases over.”
So, how did a billion-dollar startup come to bet on a tiny, unproven infra product with zero customers? There were a few factors:
First, Cursor was not yet a billion-dollar company in the fall of 2023, but a relatively fresh startup. The company raised $8M in seed funding in the same month. In 2024, the company was valued at $400M, and then its valuation surged to $29.3B. It was sold to SpaceX for $60B this year.
Secondly, it involved a tweet Simon posted after having had enough of building all summer, and wanting to test the waters. As he told me:
“I was so sick of working on [turbopuffer]. I’d been working on this all summer and didn’t know if anyone cared. I only wanted to work on it if anyone cared. So, I decided to put it on Twitter.
At that point, I had a single TMUX instance running on an 8-core node somewhere in GCP. I was thinking: “if someone goes to prod, I’ll set it up properly on multiple nodes.
But first, let’s see if anyone cares.
Anyone who’s worked in the internals of databases would’ve had too much pride to ship anything like that; I was just releasing it like a SaaS project. Why can’t you work on a database like it’s SaaS?”
Simon had the confidence to launch his product because he knew it was rock-solid and scalable:
“I knew turbopuffer was reliable. It upheld its invariants. For example, you shut down all the VMs and no data is lost. All the writes are committed directly to blob storage.”
Cursor reached out. The company was an eight-person team at the time, fresh from seed funding and growing fast, but with a search problem that they were looking to partner with a startup on. Simon recounts:
“Knowing Cursor’s founders now, I’m sure they must have sat at the dinner table one day and were like: ‘the unit economics of what we have right now, where all the vectors are in DRAM, are not working.’
They probably were asking why someone had not built a solution where you can put all the vectors from the codebase into S3 (to make it cheap), and move the part of the codebase used in memory (to make it fast). Then, everything sits in blob stores and you just hot load it all into cache. When you open the codebase, after a few seconds it’s all in RAM and the queries are as fast as everything else.
Aman (one of the cofounders) was already talking about using S3 as a key-value cache, which at the time, barely anyone was thinking of for unit economics.”
Simon knew nothing about B2B sales at that point, and was not trying to “sell” anything to Cursor. He exchanged a few emails with the Cursor team about their use case of searching many small local codebases. He wanted to help Cursor with the unit economics, while also proving that turbopuffer works. So, he hopped on a flight from Canada to San Francisco, arrived at Cursor’s office and got down to debugging their database provider:
“When I showed up at Cursor’s office, they were having some Postgres problems. I asked, ‘do you guys have pganalyze?’ And they didn’t, so I was like, ‘Okay, let’s get that going. Let’s look at it. “
And the problem was the same thing as it always is with Postgres: autovacuum hadn’t run enough. And so they had all of these going to heap, when they should be doing index scans, etc. So, we were talking about all of that.
I was just helping them: my “database genes” just kicked in. I think this built enough trust with them to believe that if I know enough to help them with their database, maybe I also know how to build one.”
It was around when turbopuffer’s other cofounder, Justine Li, joined that the work with Cursor kicked off. Simon describes Justine as the best engineer he ever worked with at Shopify, and together, they made more performance optimizations. Cursor migrated their local code-base search over to this brand new product.
Simon promised to reduce Cursor’s bill by 95%, and delivered – thanks to napkin math! Cursor was spending around $80K/month on indexing and search; with turbopuffer in place, this dropped to $4K/month. Simon was confident about making this prediction thanks to napkin math! He did the calculations based on the fundamentals; only counting resource usage that Cursor was using, and not taking a margin for turbopuffer’s operating cost at the time.
6. Reasons to raise venture capital
After securing Cursor as the first customer, Simon was not convinced that turbopuffer should raise VC money. He explained his thinking:
“I understood that if you take venture capital, no matter how many smiles there are in the room, everyone’s expecting to earn a big return on some timeline that makes sense to everyone involved. And ‘everyone involved’ are pension funds in Canada.
But at the time, I did not know if turbopuffer could be a billion dollar company. It felt like a very niche kind of search engine. And that was completely fine with me!
So, I just looked at what we invoiced and I looked at my GCP bill. And I was making this equation that:
Customer Invoices >= GCP bill [the turbopuffer costs]
Justine and I were going to optimize the system until these numbers were roughly equal. And if we could get some other workloads over to turbopuffer, we could then start paying ourselves.
But either way, I didn’t know if I could even go and raise a bunch of money. I didn’t have any relationships. I was an outsider who grew up in Aarhus, Denmark, and moved to Canada.”
A few months later, turbopuffer ended up raising a total of $700K, just to be able to hire two engineers until the end of the year on real salaries. Simon found that most VCs did not take them seriously for wanting to raise too little money, and assumed this signified a lack of ambition! Two years later, with less than $1M in initial funding raised, turbopuffer crossed $100M in annual run rate. So, there evidently was ambition!
Since our conversation, Simon has reflected on the topic of VC capital and built a mental model of six reasons that justify raising external funding:
Reason #1: Research and development. This is what turbopuffer raised for: they needed the money to hire two more engineers until the end of the year, in order to build out more of the platform. Then, they would either get more customers and generate enough revenue to not need funding, or shut down the project if it failed to gain traction.
Reason #2: Growth. When you’ve built something and want to tell the world about it, often, you have to spend money on doing that.
Reason #3: Massaging founders’ egos. Simon:
“This is a very, very popular reason! You see big numbers, you get lots of press. But I think it’s a really dangerous reason to raise money.
I wish this reason was talked about more, because you are diluting all of your employees when you do it. And for some people, it can become a status game. It’s not what we are about: we’re here to build a big business together.”
Reason #4: Employee rewards. A startup is a long journey, and everyone wants to work with the best people. But there are necessarily few of these folks, so you want to reward the standout ones by raising money so that employees can sell their shares. This was why, in December 2025, turbopuffer raised a round where they let employees sell some of their equity: it meant they would not have to wait for a future financial event like an IPO.
Reason #5: Strategic partnerships. It might be the case that raising from a VC whose network of connections is key to your business succeeding, or that taking funding from a strategic investor gives you access to their platform or services.
Reason #6: Mergers & acquisitions (M&A). Purchase another company with the funding in order to expand the business.
Simon encourages founders to be honest about why they raise capital funding – and to watch out if the reason is egocentric!
It’s inspiring to see how far it’s possible to get with “napkin math” and by understanding the bottom layers. It didn’t sit right with Simon to make decisions about vendors based on hastily written benchmarks that might measure the wrong thing. By understanding the constraints of data transfer latency and storage cost, he found a way to estimate the theoretical lower bounds of the system.
This helped lead to more informed design decisions at Shopify, and showed him there was an opportunity to build a faster, cheaper search product than the status quo.
Getting “lucky” in business requires a lot of skill, and in-person greatly helps with first impressions. Turbopuffer’s first customer being Cursor sounds almost too good to be true. But the account from Cursor cofounder Sualeh Asif – and now from Simon – reveals the ingredients in more detail:
Spotting a new business need: AI-native startups like Cursor saw their search bills explode, but needed search functionality to offer usable AI products
Offering a product with a magnitude of lower pricing: what people made suddenly pay attention to turbopuffer was the promise of not “just 20-50%” cost savings, but a seemingly radical, 90%+ reduction in costs. When you’re late to enter a market (like search), you need major differentiation, then deliver on it! This is also what Cursor’s attention.
Building trust before a sale: Simon helped the Cursor team fix their existing database before discussing using his product
In-person impressions: if Simon had not flown to San Francisco to meet the Cursor team in person, would they have taken a bet on turbopuffer?
Launching at the earliest opportunity: none of this would have happened if Simon did not announce the first version of the product when he knew it could work, but was still in a pretty unpolished state!
It’s always helpful to be aware of the dynamic of venture funding – some of which are rarely mentioned. Simon knew investors expect returns and growth on timelines which they set. Raising money is helpful in many situations, but problems arise from egotistical reasoning, and Simon believes too many founders prioritize them – knowingly or not.
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.Today, we cover:Grok’s CLI uploaded all your local files to the cloud, then got caught. Just as devs were starting to use the Grok CLI – thanks to the capable Grok 4.5 coding model – they discover…
Read more
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.
Today, we cover:
Grok’s CLI uploaded all your local files to the cloud, then got caught. Just as devs were starting to use the Grok CLI – thanks to the capable Grok 4.5 coding model – they discover…
Stream the latest episodeListen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.Brought to You by• Antithesis — with Antithesis, you can use AI agents to work on critical systems without worrying about correctness. Going far beyond code review, you can run your complete system in a hostile environment, analyze its behavior, and reproduce every issue perfectly. Teams like Jane Street, Fly.io, and the et
Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to You by
• Antithesis— with Antithesis, you can use AI agents to work on critical systems without worrying about correctness. Going far beyond code review, you can run your complete system in a hostile environment, analyze its behavior, and reproduce every issue perfectly. Teams like Jane Street, Fly.io, and the etcd community use Antithesis to ship better code, faster. Learn more.
• Buildkite —the CI platform trusted by OpenAI, Anthropic, Cursor, Meta, Uber, Ramp, Nvidia, Airbnb and many more. Buildkite was stress-tested at the largest scale inside companies solving some of the hardest engineering problems. It’s built to reliably manage whatever your coding agents throw at the build queue, today, next year and beyond. Learn more.
• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is one of their new, neat tools, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.
In this episode
Knowing how LLM contexts work and how to work around context limitations – aka “context engineering” – is becoming more important for software engineers working with LLMs. Let’s look into what works and what doesn’t, today.
In this episode of The Pragmatic Engineer podcast, I sit down with the CEO and cofounder of HumanLayer, Dex Horthy, who coined the term “context engineering”. We discuss the ideas behind this context engineering, harness engineering, loop engineering, software factories, why his approach to AI-assisted software development has evolved, and how HumanLayer is helping engineering teams automate more of the software development lifecycle without sacrificing code quality.
Key observations from Dex
Here are 12 useful points Dex made in our conversation:
1. Dex talked with ~100 “real” AI Engineers and wrote the popular ‘12-Factor Agents - Principles for building reliable LLM applications’ based on what he learned. Around August 2024, he started to build AI agents when the common approach was to use frameworks like LangChain and CrewAI, and also talked with around 100 AI engineers doing tangible work such as taking on $100K+ contracts to ship AI solutions within enterprises. They tried those frameworks and discarded them in favor of building their own pipelines. Dex shared the learnings from the conversations in his hit book, 12-Factor Agents. It’s a great read!
2. Lesson learned: Shipping unread code spells disaster within months. Dex experimented with having the model write the code and humans not reviewing anything in July 2025. Four months later, they shut things down and threw the whole system out. Production broke, and no matter how much the team prompted Opus 4.1, the model could not find the root cause.
It took days of wading through spaghetti code to discover the primary key wrongly routed through the complete codebase. Once fixed, it took three weeks to re-onboard to a codebase no human had ever read. Today, Dex thinks this problem wouldn’t even take four months to develop, now that newer models produce code a lot faster than a year ago.
3. Today’s coding models are most likely trained in a way that makes codebases worse over time. Dex believes that the reason we see LLMs “degrade” existing codebases is because they are optimized to do well on SWE-bench-style benchmarks. These benchmarks reward reproducing a known fix in codebases like Django, but cannot measure poor architecture decisions. This is because the cost function of bad architecture and bad program design cannot be evaluated by running a unit test.It’s a tricky problem to solve: Dex’s best guess is an eval of a model building 20 features in a row in a codebase without knowing what’s coming next.
4. Context engineering 101: figure out where the “dumb zone” begins. As a rule of thumb, the less of the context window that is used, the better the outcomes are. This is because the attention mechanism is quadratic: the more that goes into the context window, the more compute is required to process it all. We cover more about self-attention scalability challenges in our ChatGPT deepdive.
For a model with a 1M context window, Dex pushes it to around 300-400K when it feels right. For smaller models, he stops at around 100K. You hit the “dumb zone” when its performance starts to degrade because the context window fills up beyond this heuristic limit, and the model begins doing increasingly stupid things like deleting your .env file, for example.
5. A larger context window does not mean a smarter model. Models’ intelligence is behind the ability to use the tokens in the context window, by deciding which parts of the context are relevant for the next decision. You have to get a feel for how much context usage makes sense, and experiment when you hit the “dumb zone.”
6. Frequent, intentional compaction is a technique Dex uses for more complex projects. He will take a long and noisy context, compress it into a Markdown document, then start a new session fresh, pointing the model to this “compressed context” that is in the Markdown.
A workflow he uses:
One session reads a ton of code (filling up its context window while in the “smart zone”), then emits a research document
The next session takes tickets describing the work to be done and turns it into a design document
The following session takes both documents to create a plan
The human is in the loop where it really matters: in this case, reviewing the design document and architecture because Dex finds models to be weak on this
7. Don’t bother optimizing LLM usage until business is booming, there’s massive scale, or high costs. Dex suggests to always start building software with the smartest available model to solve the problem, since engineering time is almost always the bottleneck. Begin to optimize LLM usage and context usage only when at real scale and costs are high enough. That’s when it can be worth using GPT-OSS-120B (1/1,000th the cost of Opus) for the simpler steps in your process.
8. “You’re completely right!” or “you’re right to push back on that” are phrases that mean it’s time to start a new session. These responses mean the LLM session is trajectory-poisoned, and you’re wasting time and tokens to continue. Models are autoregressive, so if you get into this loop of:
Model makes a mistake
→ user “yells”
→ model keeps making mistakes
→ user “yells”
… the model calculates that the next most probable message is to make another mistake!
9. Only four things matter in the context window:
Size: the bigger it is, the more space you should have before hitting the “dumb zone”
Information quality. Once something is in the context window, every subsequent turn treats it as fact. This is why errors can compound.
Missing information: if there’s information missing that the agent would need, the outcome will be worse, as the agent fills in the gap with guesses.
Trajectory. Models are autoregressive, so they predict the next message in the conversation based on previous ones in a kind of thread of reasoning. “Trajectory poisoning” is when the agent gets into a pattern of doing things you don’t want. In this case, it’s time to start over.
10. Slow loops are Dex’s favorite way to do ‘loop engineering.’ The HumanLayer team started with a nightly automation setup that kicks off an agent to fix one thing in the codebase, and open a pull request. In the mornings, they woke up to a PR waiting to be merged. They tweaked it over time and now have four agents open a total of four PRs by the morning, with the focus on code quality improvements. A person still reads all of them before merging.
11. “Token harder” vs. “token smarter”: Dex is in a group chat named ‘Hyper Engineering’, where members share advice on how to max out their Claude subscriptions. This approach, he calls “token harder”. On the other side is “token smarter”: aiming to get maximum value from AI while keeping control. Smarter is harder to pull off.
12. Three ways to run a “software factory.” Here’s options Dex sees as viable:
“Turn the lights off:” go all-in on agentic coding, do not review the code, and pray that AI doesn’t create too much slop. Dex tried this and failed.
Read and review all AI-generated code. This slows things down to human speed. Dex says that this way, you should expect a 30-50% lift in productivity from AI, compared to pre-AI engineering.
Find leverage, but keep people in the loop. Find out where an hour spent in planning could save four hours’ worth of implementation, in terms of fewer bugs. Invest more time in areas with leverage: design, architecture, and key decisions. Then, let the agent generate code and don’t insist on reviewing all of it. In this way, Dex believes you can move 2-3x faster than when devs wrote all code by hand.
The Pragmatic Engineer deepdives relevant for this episode
“Loop engineering” has become a trending topic in the past month, after some high-profile folks at Anthropic and OpenAI revealed that they have stopped writing prompts, and started designing loops. At Anthropic’s developer conference, Boris Cherny, creator of Claude Code, said (emphasis mine):“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”Soon after, Peter Steinberger, creator of OpenClaw, preached loop design in a p
“Loop engineering” has become a trending topic in the past month, after some high-profile folks at Anthropic and OpenAI revealed that they have stopped writing prompts, and started designing loops. At Anthropic’s developer conference, Boris Cherny, creator of Claude Code, said (emphasis mine):
“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”
Soon after, Peter Steinberger, creator of OpenClaw, preached loop design in a post:
Elsewhere, Addy Osmani, formerly of Google, wrote an article, ‘Loop Engineering’:
“Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.”
That’s three mentions in quick succession of this new approach, which is a novel one and therefore pretty abstract to me. To find out more, I turned online to some of the folks who read these articles. In replies, you told me what “loop engineering” means to you and gave some examples of loops in your work.
Today, we cover:
Where it began: “Ralph Wiggum” loop. A year ago, software engineer Geoffrey Huntley shared how he builds “loops.” In December, the approach went viral and “Ralph loops” were born.
The /goal command ships in all major harnesses. By May this year, the major AI coding harnesses added support to run a loop from a single prompt, using the /goal command.
Loops which devs use: triggers and cron jobs. I asked devs how they use loops. Most use cases involve responding to events or running scheduled jobs. They are useful, but don’t feel like brand new workflows.
Helpful loops for devs. Open PRs for newly-recorded app issues, have notes ready when an oncall joins an outage Slack channel, “babysitting” and fixing nightly end-to-end tests, and more.
Disappointment and “tokenmaxxing”. Several devs reject looping after trying it. Agents drifting, and the “human in the loop” having better results are some reasons. Also, at companies that pay API prices for tokens, loop engineering gets expensive fast.
Was looping a hack while tooling caught up? Distinguished engineer Max Kanat-Alexander believes the “loop” might have just been a temporary hack while the harnesses added the ability to do the same from a single prompt.
Does “context engineering” matter more for devs? Except for engineers building AI infra, there seems little benefit in going deep into loop engineering. Instead, becoming familiar with AI context windows – also part of building loops – could be more useful.
1. Where it began: “Ralph Wiggum” loop
Exactly a year ago, software engineer Geoffrey Huntley published the article ‘Ralph Wiggum as a software engineer’. The name references the naive son of the local police chief in The Simpsons, who is extremely eager to always be helpful. In engineering, “Ralph” is intended to continuously nudge the agent in the right direction. Geoff described it:
“Ralph is a technique. In its purest form, Ralph is a Bash loop:
while :; do cat PROMPT.md | claude-code ; done
Ralph can replace the majority of outsourcing at most companies for greenfield projects. It has defects, but these are identifiable and resolvable through various styles of prompts.
That’s the beauty of Ralph - the technique is deterministically bad in a nondeterministic world.”
The article expands on the idea of the Ralph loop:
Start the agent with a prompt that captures the task and defines a goal
Create a plan of work, each item with a success criteria
Start the loop:
Take one item per loop
When the agent is done, check if the goal is achieved
If not: start the agent again, with a clear context window
Start loops if needed: spawn subagents as and when necessary
Geoff published the experiments he did with this approach, such as building a new programming language last summer, and said it requires skill:
“Engineers are still needed. There is no way this is possible without senior expertise guiding Ralph. Anyone claiming that engineers are no longer required and a tool can do 100% of the work without an engineer is peddling horses***.”
The “Ralph method” blew up late last year with the arrival of better models which were surprisingly capable of building ambitious projects. Software engineer Matt Pocock created a tutorial, ‘Ship working code while you sleep’ with the Ralph Wiggum technique. He said:
“One of the dreams of coding agents is that you can wake up in the morning to working code, where your coding agent has worked through your backlog. It has spit out a whole bunch of code for you to review, and it works.”
Before the Ralph loop, Matt did this in two steps:
Ask the agent to create a detailed plan for the work, with tasks broken out
Then, in a sequential order, have the agent complete each subtask in a separate run
Pre-Ralph: The agent completes one step at a time in a single context window. Source: Matt Pocock
A problem with this approach is there’s no easy way to add new tasks to the “masterplan”. Software engineers know that most plans do need to be modified, often while the work is ongoing. In contrast, with Matt’s take on the Ralph method, the “masterplan” is continuously updated in a “master PRD”. Here’s the prompt he gives the agent:
Choose the next feature: Find the highest-priority feature to work on and work only on that feature
Have tests pass: check that the tests pass (via pnpm test)
Update the master tracker: update the PRD with the work done
Log work: append your progress to the progress.txt file
Commit: make a git commit of the feature
This style of working is more of a “dynamic Kanban”:
“dynamic Kanban” style of working. Source: Matt Pocock
The “Ralph method” is all about working around context window limitations. Back in mid-2025, the maximum size of a context window was around 200,000 tokens. That’s not enough for more ambitious tasks, so it’s necessary to break up agent runs into smaller ones and run them, one by one. In this context, here’s where the Ralph method works:
Have a goal for a project, and keep running (or re-running) agents until this goal is reached
Persist work done in a “compressed” manner on the filesystem (as logs or an updated plan)
Start agents with fresh context to minimize “context rot”
Allow each agent to add or modify the “masterplan” if needed
2. The /goal command ships in all major harnesses
For a few months, building a Ralph loop meant doing it yourself: setting up the loop, state tracking, deciding how the agent can add tasks and when to stop. But things changed once coding harnesses made it easy to run these loops.
April: Codex ships /Goals
About six months after the “Ralph technique” started gaining wider traction, Codex shipped the “Goals” feature in Codex. From the documentation:
“Goals are persistent objectives in Codex that keep a thread working toward a defined outcome across turns. A Goal gives Codex a completion condition: what should be true, how success should be checked, and what constraints must stay intact.”
Also from the docs (emphasis mine):
“A normal prompt says: do this next thing.
A Goal says: keep working until this outcome is true. In a normal request, Codex works through the immediate instruction, reports a result, and waits. With a Goal, Codex has a durable target attached to the thread. After a turn finishes, it can inspect the current evidence and decide whether the objective is satisfied. If the answer is no, and the Goal remains active and within budget, Codex can continue from the latest state.”
/goal Reduce p95 checkout latency below 120 ms on the checkout benchmark while keeping the correctness suite green
That’s a clear enough “end criteria” to just hand off to the agent, which then breaks up the task, spawns subagents, and runs until complete. So, how did OpenAI build Goals? They used files, logs, running tests, and lifecycle controls:
The architecture of the Goals feature. Source: OpenAI
In this way, “Goals” feels awfully similar to a Ralph loop, except compressed into a single command! It feels like the Codex team took the idea of the Ralph loop, built infrastructure around it, took care of coordinating agents by not having them step on one another, dealt with state, running of tests, starting and stopping agents, and then added functionality such as being able to set a budget.
May: Hermes and Claude Code follow with /goal
Three days later (May 2), Hermes agent, a popular AI agent framework and OpenClaw rival, shipped their implementation of /goal. From the docs:
“[/goal] It’s our take on the Ralph loop, directly inspired by Codex CLI 0.128.0’s /goal by Eric Traut (OpenAI). The core idea — keep a goal alive across turns and don’t stop until it’s achieved — is theirs. The implementation here is independent and adapted to Hermes’ architecture.”
Less than two weeks later – on 12 May – Claude Code also shipped their /goal command. It’s identical in what it does to Codex. From Claude Code:
“The /goal command sets a completion condition and Claude keeps working toward it without you prompting each step. After each turn, a small fast model checks whether the condition holds. If not, Claude starts another turn instead of returning control to you. The goal clears automatically once the condition is met.”
A few months before, in March, Claude Code shipped the concept of scheduling an agent with the /loop command. It is basically what JavaScript’s setTimeout() function would be equivalent to: repeat a task after a given interval until the work is done.
By May, running a Ralph loop had become as simple as giving a single command in one of the major agent harnesses. It’s as if AI labs noticed user demand to do more with agentic loops which were hard to set up, and built ways to make it simpler. For instance, in open source agent harnesses like OpenCode, there are plugins like the /goal plugin. For the minimalist coding agent, Pi, the /goal command can be added as a package.
3. Loops which devs use: triggers and cron jobs
By May, we had access to the /goal primitive. So, what use cases were Boris Cherny and Peter Steinberger referring to in terms of spending most of their time on designing loops, instead of writing prompts? I asked around for examples of “loop engineering” from fellow devs. Based on ~210 replies, mostly from X and LinkedIn, it seems that triggers and cron jobs are two very common use cases for loop engineering:
Triggers / automations: an agent kicks off when an event happens. The event could be an error being logged, a new ticket created, customer support feedback received, etc.
Pre-AI, these events were typically triggered by a webhook, and kicked off things like a Slack bot posting in a channel, triggering a system, or being the starting point of a Zapier or an n8n integration.
Cron jobs: many devs see “loop engineering” as kicking off jobs that involve agents on a cadence. This is fundamentally the same as scheduled cron jobs. From director of engineering, Oded Messer:
“The idea is that strategic workflows that are repeatable and automatable can be done so with an agent. OK. But if my strategic workflow is automatable then it either becomes tactical if the AI is capable enough or it’s just a high level old-school-automation I can set up like a cron or a trigger.
The name suggests simplicity and repetition. Sometimes it feels like AI enthusiasts forgot automation was a thing before LLMs.”
4. Helpful loops for devs
Below are some workflows with AI agents that run on a regular basis as a /loop command, or when triggered:
Development-related work
Open PRs for newly recorded app issues. Software engineer Ivan Pantić:
“App encounters a problem and creates a sentry issue. Then: → Cron tells agent to check sentry and open PRs. →If there’s no active PR, agent tackles the issue and creates one → If PRs aren’t reviewed, agent pings the devs via slack
In this flow, there is only one PR open at a time.”
Fix flakey tests. Paul D’Ambra, software engineer at PostHog:
“/loop pull the next flakey test from the trunk API. run it to check if it flakes locally, if it does open a PR with the fix... which netted me 13 PRs to stabilise some of our tests.”
Triaging issues and outages. Software engineer Ivan Abad:
“A new alert/exception pops up in a channel. → The agent investigates the issue → if it is a code change, implements the change → creates the PR → pings the human for review.
Same applies for customer tickets or incidents. By the time you get paged and connect to the incident the agent already triaged everything and often located the root cause.”
Review design plans. Artem Nikitin, software engineer at Elastic:
“What I’m finding myself doing often recently is to review design/implementation plans in a loop.
Usually, agents only find a few issues during a normal run and then find more on subsequent runs.
So I’m now asking them to run in a loop until they find 0 new major issues.”
Daily/nightly work
Daily product improvements. Jack D, founding engineer at Schematic:
“We have a loop which reads the logs for the last 24 hours, user feedback and makes PR with fixes - we still review the PRs it makes though!”
Nightly end-to-end test run babysitted by an agent. Utku K, engineering manager:
“nightly e2e runs on the frontend app. When a test fails, an agent investigates first to work out whether it’s a real regression or a false negative. If it’s a real bug, the agent attempts a fix, reruns the test, and keeps iterating until it passes or hits a retry cap and escalates... Either way it lands as a PR ready for review by morning.”
More complex development work
Build new telemetry integrations and verify they work. Lawrence Jones, software engineer at Incident.io:
“A lot of the AI runbook that we use to build new telemetry integrations uses loops such as executing the query, verifying that it executed correctly and iterating on the query plan/output format/whatever until it is happy with the outcome.”
Do a long-running migration, mostly autonomously. Startup founder Rafel Mendiola:
“Recently, I converted my startup’s codebase from a regular React app to a React Native app using Expo.
There are two ways I could have done it:
In the traditional software engineering style, I could have created a large epic with 50-100 different tickets. I started building myself the infrastructure for that, it felt like it was way too much work.
What I did instead was create a skill that would let an agent figure out a small to medium-sized piece of work or piece of code to convert, given certain detection mechanisms and guidelines, and then do that conversion and keep track of the migration progress. Then I put that skill on a cron job. This was a lot easier to handle cognitively than managing a large migration plan. It also ran every 30 minutes, not nightly or weekly.”
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.Today, we cover:Bun’s Rust rewrite with Fable: what can we learn? To a sceptic, spending $165K to migrate Bun from Zig to Rust sounds very expensive. But to a realist, shortening a 1-2 year migration down to 11 days opens amazing new opportunities for devs. However, a thoroughly-tested project is required to pull it off.Anthropic’s Fable,
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.
Today, we cover:
Bun’s Rust rewrite with Fable: what can we learn? To a sceptic, spending $165K to migrate Bun from Zig to Rust sounds very expensive. But to a realist, shortening a 1-2 year migration down to 11 days opens amazing new opportunities for devs. However, a thoroughly-tested project is required to pull it off.
Anthropic’s Fable, OpenAI’s GPT-5.6 Sol, Cursor’s Grok 4.5, Meta’s Muse. Coding LLM wars heat up: Fable is back, OpenAI releases a comparable GPT-5.6 Sol, Cursor offers cheap & very capable Grok 4.5, and Meta is back with its first truly competitive coding model since Llama 3. But how did Gemini slip out of the top-ranked AI coding models?
North Korean hackers keep trying to infiltrate full-remote companies. The founder of a Canadian digital consultancy caught a North Korean dev red-handed, using an AI filter. These events are now so common that it’s hard to trust remote interviewees are who they claim.
Industry Pulse. Meta’s key logging exposed sensitive data, massive cuts at Xbox, Meta could not buy enough AI capacity from Google, Qualcomm acquires Modular, and memory price hikes hit Apple products.
1. Bun’s Rust rewrite with Fable: what can we learn?
Last week in San Francisco, I met Jarred Sumner, creator of JavaScript runtime, Bun, and was keen to learn more about the rewrite of Bun from Zig to Rust. But at the time, Jarred didn’t want to say too much, as the tool used for the migration, Fable, was out of action due to the US government imposing export controls.
Jarred and I at Anthropic’s HQ, last week
Fortunately, the situation is now resolved and Fable is available globally, and Jarred has published a detailed post about the project. Before we get into the migration, some context:
Stream the latest episodeListen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.Brought to you by our presenting partner, Antithesis.Verify your system’s correctness by running your whole system in a hostile simulation and finding bugs. I’ve been using Antithesis myself, and I’m impressed with their innovation in building new kinds of debugging tools. Like this:Bug probability analysis in Antithesis’ f
Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to you by our presenting partner, Antithesis.
Verify your system’s correctness by running your whole system in a hostile simulation and finding bugs. I’ve been using Antithesis myself, and I’m impressed with their innovation in building new kinds of debugging tools. Like this:
Bug probability analysis in Antithesis’ fully deterministic, simulated environment. When probability spikes, it’s a good place on the timeline to “rewind” and check logs to find what triggers bugs.
In this special “ask me anything” episode of Pragmatic Engineer podcast, I am in the hot seat facing questions sent in by subscribers that are read out by guest Volodymyr Giginiak, CTO and cofounder of Wordsmith AI, a legal tech startup (note: I’m an investor).
I tackle your questions on the software industry, AI, hiring, engineering organizations, career growth, the business model of the Pragmatic Engineer, and more. We also discuss where software engineering is headed, and I offer advice on some specific situations. Thanks to everyone who sent questions!
Three stories & observations
Story #1: Without the COVID-19 pandemic, The Pragmatic Engineer might not exist. Prior to the global health crisis in 2020, I had no plans to get serious about writing: I enjoyed blogging on The Pragmatic Engineer blog, but intended to remain an engineering manager or software engineer for the foreseeable future.
But then, COVID-19 happened and Uber made layoffs, which led to a quarter of my team being let go, while the rest of us were disbanded into other teams. It was a tough time, and I decided it was a good moment to exit and finish writing a book I had been working on, ‘The Software Engineer’s Guidebook’. After that project was complete, I planned to try and start a VC-funded startup and build something around platform engineering; possibly a system for tracking RFCs at mid-sized and larger companies.
I gave myself around eight months to finish the book, but when that deadline elapsed, it still wasn’t ready. I did write three other books (‘The Tech Resume Inside-Out’, ‘Building Mobile Apps at Scale’, and ‘Growing as a Mobile Engineer’) and still wasn’t convinced by any startup idea. But I did discover that I like to write!
Story #2: I was on track to publish a damning exposé, until one message from an engineer changed my mind.
During the first year of The Pragmatic Engineer, I wrote about conditions for engineers at the Dutch neobank Bunq, based on accounts from disgruntled employees. I had a final draft ready, which I sent to the company to provide a right of reply ahead of publication. Then I received a message.
It was sent by an engineer originally from the Middle East. They told me they had really wanted to break into the European tech industry, but that no company would sponsor their visa, except Bunq. Yes, the company was a tough place to work at as a dev – and this engineer subsequently left for another opportunity – but they appreciated Bunq for taking a chance on them that enabled them to move to Amsterdam and learn how to build fintech with a small team. This engineer now works at Meta and attributes their success to the break Bunq provided.
Based on that interaction, I opted not to publish the article, and it also led me to adopt a new editorial policy that I have followed since: I write about what works inside companies, instead of focusing on what seems to be broken.
Story #3: Being called a “nobody” by a CEO led to my sole investigative piece, which uncovered some pretty interesting details. After I briefly noted Pollen’s poorly-handled layoffs, CEO Callum-Negus Fancey dismissed my report during a company all-hands, and compared The Pragmatic Engineer unfavorably to the BBC as just some minor publication with an agenda against Pollen (why I would have an anti-Pollen bias isn’t clear to me!) To be honest, I took it personally when I heard a recording of this sent to me by employees there, and so started digging around.
I discovered unpaid salaries, silently cancelled health insurance in the US, and the CTO deliberately triggering a $3.2M double charge to customers and never publishing a postmortem, despite engineers requesting one.
It was certainly something, and I published the findings in the article Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid - Exclusive. To ensure the CEO saw the report on a platform he deemed worthy, I also contributed to the BBC’s documentary: Crashed: $800M Festival Fail, aired in the UK during prime time. By doing all this, I also learned that investigative journalism is just not for me.
Opinion #1: I believe LeetCode-style interviews will stay because they self-select tolerance of corporate nonsense. The existence of data structures and algorithm (DSA) interviews is a bit of a head scratcher because these skills are rarely used at work. However, a candidate who’s willing to grind for weeks or months to prepare for an interview which bears little resemblance to the job, is likely to be someone who understands that sometimes it’s necessary to do pointless work.
This suggests they’ll probably have a much better time in Big Tech than someone who refuses to engage with meaningless tasks. It’s one reason I’ve observed for companies retaining LeetCode-style interviews. Of course, AI solves the puzzles with ease these days, and I expect larger companies to move back to in-person interviewing – all while keeping DSA interview questions.
Opinion #2: MCP became industry standard partly because Anthropic wasn’t a threat – but it couldn’t pull this off today. When MCP launched in November 2024, Anthropic wasn’t yet considered the leading AI lab. GPT-4o was seen as the top-performing multimodal model, followed by Claude 3.5 Sonnet and Gemini 1.5 Pro. At that point, Claude 3.5 Sonnet was seen as the best coding model, but it wasn’t understood how advantageous being good at coding would be for AI in general.
Therefore, OpenAI, Google, Microsoft, and other players could adopt MCP without fear of lock-in, as it came from a promising, but not a dominant lab. When Google launched its Agent2Agent protocol a few months later, no major lab adopted it due to concerns about Google’s dominant position. Today, Anthropic is the leading frontier lab and I reckon this would discourage adoption if MCP was launched in the present climate, for the same reason as nobody adopted the Agent2Agent protocol.
My answer to subscribers with questions about how to create standards is that I see them as emerging somewhat coincidentally, following a technically strong approach and with the right external conditions in place, which are impossible to entirely predict or control.
Opinion #3: My hot AI take: AI doesn’t make work easier, and be mindful of skill atrophy. If you’re using AI and life seems to be getting a lot easier, it raises the question: are you trying hard enough? Personally, using AI forces me to think just as hard, or even harder than before. I choose to use zero AI in my writing for the Pragmatic Engineer, and Grammarly is turned off as well. This is because I don’t want my writing skill to degrade, and would like to keep improving. On the other hand, with coding, I do use AI and accept my hand-coding ability will unavoidably degrade. My tip is to be mindful of the tradeoffs inherent in AI, and to keep using those skills which you value and want to keep sharp, even when AI tools are available.
The Pragmatic Engineer deepdives relevant for this episode
What is the tech jobs market like for job seekers and hiring managers today? It’s a broad question for which one answer is that it’s a land of contrasts and confusion, and also some crossed wires. Experienced engineers and managers feel ghosted by employers and recruiters, who in turn have given up on inbound applications because their inboxes are full of AI slop, sometimes from bogus candidates. It’s rosier for those with specific skillsets, who are in strong demand – and personal networks help
What is the tech jobs market like for job seekers and hiring managers today? It’s a broad question for which one answer is that it’s a land of contrasts and confusion, and also some crossed wires. Experienced engineers and managers feel ghosted by employers and recruiters, who in turn have given up on inbound applications because their inboxes are full of AI slop, sometimes from bogus candidates. It’s rosier for those with specific skillsets, who are in strong demand – and personal networks help more than ever to land the right job.
For this final part of our series on the tech hiring market during the first half of 2026, I spoke with more than 50 hiring managers, software engineers, and engineering leaders. Thank you to everyone who contributed!
For more on this topic, check out our analysis of what the data says about the market in Part 1 and Part 2 of this mini-series.
Overall, an appropriate description of the employment market as many folks experience it right now would be “weird”. This is a characteristic that’s not easy to see in the data, but is clear from talking to people and hearing their anecdotal, personal accounts of job hunting this year. I think the data in the first two articles of this series failed to capture just how unusual things are.So, in today’s issue, we attempt to shed some light on the weirdness, covering:
“Catch-22:” nobody finds each other. Hiring managers struggle to find experienced folks, who barely get any replies when applying for jobs. How’s that work?
No trust. Is AI to blame? AI-enhanced resumes read as incredible, but hiring managers often face disappointment. Some places don’t bother reading inbound applications as a result.
Hot market for some, but tough for most. For those in AI Engineering, ML, or FDE, the market is incredible. For everyone else, it’s much less great.
Higher hiring bar & lower compensation – but not for everyone. Many candidates are unhappy with offers that are the lowest in years. This doesn’t apply to AI Engineering positions or at AI businesses, however.
Engineering leader recruitment: also weird for senior ICs. Senior engineering leaders are struggling to find opportunities, or may turn them down in favor of fractional roles or to work on their own startup.
US market trends. Folks experiencing the “best market ever” are likely in the US, where a talent shortage is a bigger complaint than it is elsewhere.
Trends in the UK, EU, and rest of the world. “Ghosting” is more commonplace than in the US, “fake applicants” a bigger issue, remote roles are going extinct, and more.
Interview preparation signups: what do they indicate?
Where engineers go after Big Tech
1. “Catch-22:” nobody finds each other
The phrase “Catch 22” refers to a paradoxical problem, whose solution is blocked by the problem itself. The term originated in a famous World War II novel of the same name, and it also describes pretty accurately what I see in today’s tech job market. Hiring managers are saying that highly-skilled talent (typically senior+ engineers) is not available to be recruited, at the same time as experienced, proven professionals find their applications ignored by employers.
What seems paradoxical here is how both can be true. It’s as if recruiters and potential candidates aren’t hearing each other. Of course, there’s some nuance:
“We get about 1,000 applications a day on inbound and maybe two of them are even relevant to the posting.
I mostly no longer look at inbound seriously because it’s so c***. I’d almost certainly miss a great inbound submission if it came in.
All of our recent hires have been via network and us reaching out to folk on LinkedIn. Biggest hurdle we have to outreach is thin LinkedIn profiles and little other online presence.”
From the other hiring managers we talked with, more themes emerged:
More inbounds than ever & also more noise
“I’ve never seen so many inbounds and strong resumes. I’m hiring for lots of roles; for one software engineering position in Seattle, we have had 800 resumes inbound over a three-month period. I’ve never seen anything like this! These resumes are not low-quality either: they are people who have worked at MSFT, AWS, other large tech companies, and have solid skills.” – Head of Engineering, Series B startup, Seattle, US
“It’s difficult to find good candidates among all the noise. Have had two open senior engineer positions for months, and the only good interviews or offers we’ve extended were in network.” – Engineering Manager, late-stage startup, US
“A glut of vastly underqualified people completely drowning your hiring pipeline. This is what I’m seeing, and it’s making inbound a useless channel.” – Engineering Director, Big Tech, US
“From what I hear from recruiters, every job posted has 1,000+ applicants, and 98% of them are considered unqualified.” – fullstack engineer, Middle East
“We are a small company and gave up on inbound hiring. We got too many AI applications. Much lower signal-to-noise compared to past experience.” – Director of Engineering, Canada
Experienced engineers struggle to get interviews
“Even when you’re good, it feels like there’s a LOT of noise to cut through to get noticed. I’m quietly looking for a new role in devtools, and fit that ‘product engineer’ profile perfectly. I feel like the demand for great people is higher than ever, but I can’t figure out where on that bar I fall.” – Tech lead, seed-stage startup, US “As an experienced engineer & successful-ish founder I have yet to get to a phone screening in this job market. There is clearly something amiss, just a wall of noise preventing any signal from getting through.” – Software engineer + founder, 8 years experience, US
A staff engineer at a late-stage startup in the US summed things up:
“It feels like everyone who has a good job is holding onto it for dear life, and THOSE are the people we want to hire.”
“Tale of two cities:” in demand or not
A director of engineering at a Big Tech in the US, identifies two distinct groups in the market:
“If you are at the top of your game, have AI experience, and are senior enough, you can write your own ticket. If not, then the job market is tough! This job market is like the tale of these two very different cities.”
Observations from some job seekers echo this: demand feels strong for “AI-adjacent” engineers (those building AI systems), especially in the US. Standout engineers with a strong network are still in demand, and we covered the story of one such person in “How to be a 10x engineer” – interview with a standout dev. For everyone else, it’s a struggle to get an interview!
Referrals: more important than ever?
Several experienced engineers currently on the job market say they only get interviews when they personally know someone at a company:
“Larger companies have to have some serious AI going on to sift through resumes because they are being bombarded. It really is the case that you don’t get your resume seen unless you know people who can vouch for you.” – Technical Program Manager, Big Tech, US
“Referrals are a lifeline. It’s impossible to get interviews for Staff or Principal Eng positions by cold applying. The only interviews I am getting from a cold-apply are Senior-level roles.” – Principal Engineer, 10 YOE, US
From a couple of hiring managers:
“The only interviews or offers we’ve extended were to devs already in our network.” – Software architect at a mid-sized company, US
“I filled an engineering manager position in record time thanks to a person in my network looking for this job. If it would not have been for my network I would probably have been looking for someone for a couple of months.” – VP of Engineering, private equity-funded company, Germany
2. No trust. Is AI to blame?
Hiring managers repeatedly tell us they no longer trust what they read in resumes, and that some candidates even turn out to be fake.
Polished CVs, weak candidates
A common gripe among hiring managers is that resumes are highly optimized by AI, but candidates turn out not to have the experience they claim:
“CVs are high-quality, but the people behind them are not. Almost every resume looks impressive. However, the quality of the conversations does not match it at all.
One recent example: I interviewed a senior candidate who had spent five years at a US-based cloud consulting company, most recently as an architect. I asked which architectural principles or patterns he had used in his projects. His answer was: “Daily standup, sprint planning, and retrospective.” I clarified that I meant from a tech perspective, not process perspective. He confidently replied: “Yes, daily standup, sprint planning, and retrospective.” – Engineering manager, large company, Berlin, Germany
“Ten years ago, we would have appreciated resumes tailored to a role that was posted, now it’s just lazily thrown together with AI.” – Engineering Manager, mid-sized company, Canada
AI-keyword stuffing is rampant, according to one head of engineering in the UK:
“Lots of people are rebranding themselves as senior AI Engineers and demanding much higher salaries. Their resumes now have lots of AI-related keywords mentioned, like RAG, evals, inference… but when digging deeper there is little substance. Many of them are seeking a senior level salary (£90k–£140k) when they are barely showcasing mid-level skills.”
Cover letters are as good as dead, several hiring managers tell us. The reason is that they’re always AI-generated, boring to read, and pointless. More than 18 months ago, we reported on how cover letters were being made redundant by AI in How GenAI is reshaping tech hiring.
An engineering manager at a UK e-commerce agency summarizes what’s happening:
“’Claude; write me a CV that matches this job spec, then auto send’. This seems like the name of the game for most applicants.”
Fake candidates
Does anything encapsulate the challenges of today’s job market for recruiters more succinctly than candidates who do not exist, even when they appear to be sitting in an interview? A year ago, we covered an “AI faker” applicant caught by a security startup, who was potentially a state agent from North Korea.
Catching an imposter: candidate (left) refuses to place their hand in front of their face because it would blow their AI cover. (right) The interviewer illustrates the request. More in our deepdive
Such incidents are becoming more common at US, UK and EU companies which hire for remote roles:
“There are a lot more fake candidates applying, leveraging AI for not just resumes, but also interviews. In extreme cases, the interviews are being outsourced so that a different person shows up for the interview. It feels a bit like playing captcha with them during interviews.” – Senior EM, private-equity backed company, Bay Area, US
“The second person we interviewed was clearly a North Korean scammer, writing questions into an LLM, reading the response, easily tripped up, and other interviews were background noise in the room.” – Staff Engineer, UK
“Many applicants that looked a good fit turned out to be someone else in the Asia Pacific region doing interviews with an AI in the background. It’s easy to spot because of the ‘lag’ in a naturally flowing conversation.” – head of engineering, mid-sized company, Germany
Cheating in remote interviews by using AI is also commonplace. A software engineer based in Finland told us:
“A couple of times we saw an applicant who was using AI to answer the questions, and this was sooo weird. I honestly didn’t know how to react, so we decided to cut the interview short. We then shared a note with the recruiters on how to spot this.”
3. Hot market for some, but tough for most
The market appears sharply bifurcated: amazing for AI and a few specialist roles, and a major struggle for everyone else.Some accounts from folks benefitting in the current conditions:
“The market is really good as an (AI) engineer with the right experience. I’m not actively looking but get 2–3 messages a day. When I was hiring for AI engineers at my last AI startup, the market for these folks was terrible. It’s very hard to find good talent, and people that were great on paper did poorly in interviews.” – AI Engineer, 5 YOE, New York, US
“It seems like demand for senior positions is still there, but only if your profile really matches what the company needs. For example, I’ve been working on different data pipelines for a while, and finding such a position is relatively simple. However, breaking into anything else is not! Every startup wants to hire only people experienced in that particular thing they need.” – Software Engineer, 13 YOE, Big Tech, UK
Here’s what two job seekers who are struggling at present say:
“I sent my handcrafted resume to 30–40 positions, and heard back from zero. Eventually, I got an interview after a recruiter reached out via LinkedIn; I don’t know how I would’ve found anything, if it was not for this!” – Software engineer, 5 YOE, Amsterdam, Netherlands
“As a developer without a specific “specialization,” I’m struggling to get any interviews. I’ve spent a few years as frontend, then another few as backend, and most recently working on DevEx. In this market, I just don’t get any callbacks for interviews, not even a first round interview.” – Software engineer, 6 YOE, London, UK
Anecdotal evidence from job seekers suggests that being in these fields means the market is as good as it gets, right now:
“It’s the greatest job market I’ve ever seen. I’m an L5 former FDE, now SWE, who has worked on LLM apps for ~2 years. The inbound top of the funnel is bonkers, and I find myself saying “no” to places that I would have once killed to work at.” – AI Engineer at an AI decacorn, San Francisco, US
“I have seen huge interest in FDEs and AI Engineers as an engineer who’s been interviewing.” – Software Engineer, 5 YOE, New York, US
“It’s not hard to get interviews as an ML/AI engineer. The technical bar is about the same as late 2022. If anything, onsites seem to be of fewer rounds for the same level. Companies seem to move fast to get candidates onsite.” – ML engineer, ex-Meta, Bay Area, US
An engineer at Apple is surprised by the demand for their skills from AI companies:
“Got a lot of responses from cold applications for AI roles, and ended up with two offers at AI infra companies. I ended up getting a significant pay bump beyond what Apple offers for the same level!”
A software engineer at well-known company in the Bay Area, also finds the job market is good:
“I got offers for Senior SWE with Stripe and Rippling, was rejected from Snowflake… Anecdotally, the job market is better compared to 2025 for regular SWE positions in the Valley.”
EM and Staff+ profiles are near-impossible to fill
A repeat complaint from hiring managers in the US is how challenging it is to recruit solid engineering managers and Staff+ engineers:
“It’s extremely difficult to hire EMs and Staff+ engineers. It’s much easier to hire folks with less than 10 years of experience. This is despite us offering 90th percentile comp via Pave, and having a hybrid and good culture.” – Fractional VP of Engineering, Series B, New York, US
“My team has been looking for a new engineering manager for three months and we barely had any good applicants. It’s a super weird dynamic right now.” – Senior Infrastructure Engineer, Series D Fintech, company with offices in the US and EU
“The most desirable candidates right now are EM or staff-level folks who can keep up with uncertainty in the business. We are having trouble hiring folks who roll with the constant change that is now very typical at startups.” – CTO at a San Francisco-based startup
“Specialist” profiles are hard to recruit
A few examples:
Distributed systems engineers: “We can’t find enough qualified distributed systems engineers, cloud infra etc. I have 20 reqs [open positions] right now.” – Recruiter at a hyperscaler, US
Product engineers: “‘Product engineer’ has been a hard profile to find. It is also hard to find someone with a decent design eye who can also build full stack. The hardest thing to hire for has been taste + trust… I’d rather hire someone who is ‘behind’ on AI, but has great taste/judgment than someone with complex agent setups and prompt libraries.” – Tech Lead, seed-stage startup, Los Angeles, US
Senior engineers, in general: “As a hiring manager trying to hire seniors right now, it has felt pretty difficult. Most applicants are totally unqualified, and the qualified ones do poorly in an interview.” – Engineering manager, robotics company, Canada
Silence for many
One in five respondents detail how difficult it has been to get responses from recruiters:
“Things are looking bleak. Now, applications are going into a void. Recruiters aren’t reaching out immediately as before. Of 10 or so applications sent out, I’ve gotten one interview scheduled that was then rescinded because it looks like the team was laid off.” – Frontend engineer, 10+ YOE, Southeast US
“The job market has been a desert. I’ve gotten back into applying, job agencies seem to have few roles they’re looking to fill. The companies I have applied to myself, never reply.” – Software Engineer, 8 YOE, Southern California
“Recruiters are ghosting me! Every time when I respond to recruiter reachouts, they ALL ghosted me after a few messages and have similar stories from friends.” – Software Engineer, 7 YOE, ex-Meta, Switzerland
“So far, I have a hard time even getting past the hiring manager and I’m rejected before I can do the proper technical assessment.” – Software Engineer, 7 YOE, Helsinki, Finland
Even folks at Big Tech face a lack of response. Here’s a Technical Program Manager currently at Meta who used to be very in-demand:
“I put a few feelers out when Meta announced layoffs, and it’s just radio silence. I don’t want to sound cocky, but with my resume, you at least get a chat with a recruiter, usually right away. But Google and Anthropic just ghosted me.”
4. Higher hiring bar & lower compensation – but not for everyone
In a “normal” market, when the hiring bar goes up, so does compensation. But we heard anecdotes about the hiring bar going up, with the compensation on offer trending down!
Stream the latest episodeListen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.Brought to You by• Antithesis. If you’re using agents to code, the problem isn’t writing the code but making sure that nothing broke. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. Teams like Jane Street, Fly.io, and the etcd community use age
Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to You by
• Antithesis. If you’re using agents to code, the problem isn’t writing the code but making sure that nothing broke. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. Teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster, with Antithesis. Learn more.
• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started. • turbopuffer. A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. It’s never been better to try it out: last week, they dropped their base price from $64 per month to $16 per month. Give it a spin now.
In this episode
Few have made as big an impact on software engineering as this week’s guest on the Pragmatic Engineer podcast, Kent Beck. He created Extreme Programming, pioneered test-driven development (TDD), co-created JUnit, and is one of the authors of the famous ‘Agile Manifesto’. But these days, he’s re-examining many ideas for the age of AI, and says we’re failing to accumulate trust during this new era at the same high rate as new code is being accumulated.
Today’s episode is the previously untold story of Kent’s remarkable career and how he became the industry legend he is today. We start with Kent’s journey from discovering Smalltalk in the early days of personal computing, to helping define modern software engineering practices. We explore the origins of TDD, design patterns, Extreme Programming, and Agile – along with some lessons learned at Apple and Facebook.
Kent explains why he believes software engineering is about far more than writing code, why no one yet knows exactly how engineers should work alongside AI agents, and how his “explore, expand, extract” framework can help engineers navigate major technology shifts.
Key observations from Kent
Here are 12 rarely-told stories and observations from Kent:
1. Coding is only a small part of software engineering – the rest can’t be automated. Kent rebuts the claim that coding – and eventually the whole software engineering craft – will vanish. He believes coding is only part of what we do, and a small part of it, too. Through your work, you also build confidence, make connections with other people, and develop your personal understanding of the domain. This will remain important, even if we develop our skills in other areas than coding.
2. Lesser-known story: fired by Apple. Kent joined the tech giant in 1987, drawn in by Smalltalk, whose language was taking off at the time. Xerox had handed the language to a few companies to see who could run with it, and Apple was one. But Apple’s customers wanted C and Pascal compilers instead, so the Smalltalk project went nowhere. Kent joined a team building a programming language for kids, but says he was eventually fired because he was in “punk mode” and wanted to do things his own way, instead of being a team player.
3. Kent used to keep a thesaurus close at hand. When working at Tektronix in the late 1980s with Ward Cunningham (the creator of the first wiki, and a pioneer in design patterns), they built HotDraw, a graphical editor with a boxes-and-arrows model. For naming core abstractions, Beck chose “drawing object,” “drawing handle,” etc. But Ward cared intensely about nomenclature and hunted for better vocabulary, so they kept a physical thesaurus which was well thumbed. Beck recalls they became pretty obsessed about the names of things.
4. TDD wasn’t ‘invented’ so much as ‘rediscovered’ by Kent. As a kid, Kent read one of his father’s programming books from the tape-to-tape era, when an input tape like time cards was fed through a payroll program to produce an output tape, such as one with checks. The book’s advice was to take a real input tape and manually type the expected output tape before writing the program. He read this, didn’t understand it, and forgot about it.
Years later, Kent built SUnit, a small testing framework, and randomly remembered the input-tape trick, so mapped it onto SUnit. If he followed the pattern, he’d write the test before the code. He laughed out loud at this because it seemed like such a stupid idea: why write a test that’s guaranteed to fail, when the classes and methods aren’t even defined yet? But when he did, he found his anxiety about programming vanished. This is when he became a TDD convert.
5. Kent invented Extreme Programming (XP) while on the Chrysler payroll project. Kent threw away a codebase that didn’t work and restarted the project with a new methodology. He paired with others, and used his own ideas for testing. Later, he coined the new methodology’s name by deliberately picking one that he knew would be unpopular with the tech establishment of the day: “extreme programming” was born.
6. The Agile Manifesto came together in a messy way. In 2001, a loose group of folks who rejected waterfall development gathered at Snowbird, Utah, to rethink programming. Kent recalls this summit proceeded badly as everyone pushed contradictory ideas. During a break, Martin Fowler and Jim Highsmith stayed behind, and when the others returned, they found the values written on the whiteboard. Kent’s contribution was the word “daily”: “Business people and developers must work together daily throughout the project.”
7. Calling it “agile” was an error. Kent objected to the word “agile” at the time, and still does today, since nobody claims they prefer “rigid” development, and everyone says they’re “agile”, even when they’re not. He would’ve preferred a less spacious term, like with “extreme programming”: after all, it’s hard to call yourself an “extreme programmer” without actually following that methodology.
8. The Dotcom Bust hit very hard. The day before the 9/11 terror attack in 2001, Kent had eight months of consulting work booked at top rates and was finishing work on a house in rural Oregon. The next day, everything was canceled, just as big bills fell due. He burned out into depression and was left unable to program for years, in what was a “lost decade.”
9. At 50, Kent joined Facebook and realized nobody cared about testing or TDD. At Facebook, Kent found a company that barely did any form of unit testing, while running a massive, stable, and fast-growing site. He signed up to teach a TDD class at a hackathon — he wrote the book, after all! The classes either side of his in the schedule both filled up, but the TDD class got zero signups, not even a pity one. He made the decision to forget everything he knew and to relearn software engineering as it was at Facebook. In the end, he stayed seven years.
10. Building software products has three phases: explore, expand, extract. This is Kent’s “3X” model. ‘Explore’ means trying many cheap uncorrelated experiments, ‘expand’ involves focusing on the one thing that’s working and overcoming obstacle after obstacle, while ‘extract’ is a repeatable playbook and economies of scale. How you code, hire, and organize differs across each phase.
11. Kent has always been an anxious programmer. He describes himself as chronically anxious because the more complex the code is, the more he knows it could break. This was the fuel behind testing and TDD, which are approaches designed to soothe an anxious mind.
12. Kent sees himself as a “tree shaker, not a jelly maker.” He starts things like patterns, SUnit, JUnit, TDD, XP, 3X, then pushes them until they take off, before moving on to the next thing. It’s his defining trait, and may explain his enormous output, and also why he abandoned TDD just as it peaked.
+1:The human part is the most important one in software engineering. As Kent explained:
“This is the biggest cosmic, practical joke ever. As young people, we were promised: “Okay, here’s this computer and once you’ve completely understand this computer, you’ll be fine. That’s all you need to do.”
So I set out the first part of my career just to become the best programmer that I could be because that’s what it would take to be successful. And then you realize: sorry, there’s this whole human side. Your ability to affect change in the world is gated by your ability to communicate with, to soothe, to understand other human beings. And those are exactly the skills that I thought I didn’t need to learn!
So I was promised: just understand the computer and you’ll be successful. And then someone went “just kidding, understand people!” And now I was in a position of being ten years behind.”
The Pragmatic Engineer deepdives relevant for this episode
Scheduling note: this week, I’m in San Francisco at the AI Engineer’s World Fair, so there won’t be an edition of The Pulse on Thursday. However, tomorrow (Wednesday) there will be a special podcast episode – the lengthiest, most detailed one yet – with software engineering legend, Kent Beck.In recent days, I’ve visited the offices of OpenAI, Anthropic, and Cursor, in San Francisco. Onsite, I talked with software folks working on the model side to learn more about how their way of building softw
Scheduling note: this week, I’m in San Francisco at the AI Engineer’s World Fair, so there won’t be an edition of The Pulse on Thursday. However, tomorrow (Wednesday) there will be a special podcast episode – the lengthiest, most detailed one yet – with software engineering legend, Kent Beck.
In recent days, I’ve visited the offices of OpenAI, Anthropic, and Cursor, in San Francisco. Onsite, I talked with software folks working on the model side to learn more about how their way of building software is changing. This article is based on observations from those visits, including some new developments that I reckon may be adopted industry-wide.
We cover:
Next mega-trend? Agents running in the cloud to go mainstream. OpenAI, Anthropic, and Cursor are all-in on cloud agents and expect demand for them to increase massively.
Mass adoption of coding harnesses by non-developers. At OpenAI, more than 95% of non-engineers use Codex, not ChatGPT. Is it a sign of things to come across tech?
Will the main task of engineers be to make agents more efficient? Ever more engineering work is about building environments for agents to execute more efficiently at Anthropic and Cursor.
Next trend? Companies aggressively optimize spend-per-token. AI spending by software engineers is so high that it makes sense for platform teams to slash per-token cost. A case study from Coinbase.
1. Next mega-trend? Agents running in the cloud to go mainstream
Last week, Andrej Karpathy employed the phrase “new paradigm” to describe using Claude Tag – a way to mention Claude in Slack and have it kick off tasks – to work with AI:
There was plenty of pushback against this claim on social media; after all, it’s just a Slack integration with Claude, right? I also thought this until I asked David Hershey at Anthropic’s Applied AI unit about it while visiting the company’s offices. He explained in detail what makes this particular Slack integration different from using something like Claude Code:
No additional setup. For Claude Code to work well, it should be connected to internal MCP servers, with the right skills on your local machine. Of course, at larger companies this setup is at least partially automated, but devs often need to do tweaking.
No “tool context-switching.” Just mention it in Slack! Of course, opening Claude Code is not a big effort, but it’s still more work than just typing it out in Slack, and kicking off work.
Routine work made easier. David has “Claude playing Pokémon” as his side project. Every time a new model comes out, he kicks off a run of his script on it. Previously, this took a few minutes to set up every time, and then it ran on his machine for hours. With this new Slack integration, it’s just one command.
My sense is that the excitement here is less about the Slack integration itself, and more to do with the fact that it’s easy to kick off one or more AIs that no longer run on a local machine. You can skip the setup entirely.
‘Claude Managed Agents’ is a big focus at Anthropic. While there, I met Katelyn Lesse, head of engineering for Claude Platform, who explained that Claude Managed Agents is a large, complex project which her team built over a six-month period. It’s a hosted service to execute long-running agents on various cloud providers.
Cloud agents are the “big deal”, not the Slack integration
Also last week, I had the opportunity to attend a private AI builders event, where Peter Steinberger discussed his workflow.
Peter Steinberger covers how he uses AI coding agents
He talked about how he has gotten really tired of having several OpenClaw agents running on his local machine, which heat up the CPU and slow down his whole system. So, he built Crabbox as a way to run OpenClaw agents in the cloud:
Crabbox: remote agents for OpenClaw
Suddenly, the same solution of cloud agents has emerged in separate places – at Anthropic and with Peter’s OpenClaw – in response to issues caused by locally-running agents. I also learned that cloud agents are becoming a big deal at OpenAI and Cursor, too.
OpenAI bets big on Cloud Agents
OpenAI acquired Ona, (formerly Gitpod), the leader in cloud development environments (CDEs). Back in 2021, CDEs were built for developers to develop software faster, and they also happen to be the perfect primitive for agents to run in a sandboxed cloud environment. From the acquisition announcement by OpenAI (emphasis mine):
“As Codex becomes more capable, its most valuable work is unfolding over hours or days, rather than minutes. We believe people should be able to delegate more ambitious work without remaining tied to the machine where it began. The work should continue beyond the initial session, with Codex making it possible to stay connected and check progress, provide direction, make decisions, and review results from anywhere.
Ona will help us do that. Its technology provides secure, persistent environments where agents can access the tools, systems, and context they need to make progress over time.
By bringing Ona to OpenAI, we will expand Codex beyond work tied to a single device or active session and help more organizations deploy agents securely in production.“
At OpenAI’s offices, I asked engineers there if their focus is shifting to cloud-based agents. Their answer: it very much is. This is a fairly recent development and they’re hiring engineers for the Cloud Agents team. Here’s one job ad that’s currently live:
“We are looking for an experienced software engineer to help build and scale our cloud agent platform. You will design and operate systems for orchestrating agents at scale. You will work closely with product engineers on ChatGPT, API, and Codex to define the right abstractions and enable them to ship products quickly. Strong backend or infrastructure experience is important; experience with Python, Rust, distributed systems, cloud infrastructure, or product platforms is especially helpful.”
Cursor: running agents in the cloud is the future
At Cursor, I spent an hour with cofounder Sualeh Asif (formerly the CTO, now Chief Product Officer). Cursor released Cloud Agents at the end of last year, and is starting to focus a lot more on this area. Sualeh revealed some interesting details about working with cloud agents:
Agents in the cloud don’t have a way to “complain.” With running an agent locally, when it gets warnings or errors, it surfaces them to a human in its response, who instructs it to do X or Y. However, there’s no such loop for a long-running task on the cloud! Cursor came up with the idea for the model “confess” in regular interviews, and the “confessions” are shared with the infra team to improve the agents’ environment.
Long-running agents have their own challenges. What happens when a node terminates, midway through; how do you move agent execution from one node to the other? There are new, nontrivial engineering challenges the team needs to solve.
Only yesterday, (Monday, 29 June), Cursor launched its iOS app that enables the building of software from anywhere.
Building software on a smartphone needs cloud agents. Source: Cursor
This product is built on top of cloud agents to allow for long-running tasks, the company said:
“Cloud agents run in isolated virtual machines with full development environments to test, verify, and demo work. Since they operate asynchronously with their own tools and resources, cloud agents can run for longer and iterate toward merge-ready PRs without intervention.
To take advantage of these capabilities, send a local plan to a cloud agent or move active agents to the cloud to keep running. You can move the cloud session back to your computer to test changes locally before merging”.
Why are cloud agents suddenly a thing?
It figures that running AI agents in the cloud is practical: there’s less setup involved, several can run in parallel, and the cloud is a better, more convenient place for long-running agents than a personal laptop is; i.e., having to keep the lid open even when walking around the office.
But why is this happening now? My hypothesis is that a mix of factors are at play:
Coding models got ‘good enough’. Before Opus 4.5 / GPT-5.4, AI models could not really code autonomously, so running them for long tasks was pointless!
Infra for AI coding agents has matured. Ways of giving more context to agents have improved: things like MCP and skills became mainstream and better understood.
The context window is bigger. Today’s models have context windows of up to 1 million tokens, meaning that more complex instructions, code, and context can be passed in. It’s hard to have agents run for a longer time without access to a large context window.
Cloud providers have much more GPU capacity. Every cloud provider has been building GPU clusters in the last few years, and now there’s enough that these AI agents can make use of this infra.
2. Mass adoption of coding harnesses by non-developers?
At OpenAI, I also met Andrew Ambrosino, who was the first engineer on the Codex team. Our time together got off to an ideal start, with Andrew saying he needed to show me something incredible:
Stream the latest episodeListen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.Brought to You by• Antithesis – If you’re using agents to code, the problem isn’t writing the code but making sure it didn’t break anything. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. With Antithesis, teams like Jane Street, Fly.io, and th
Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to You by
• Antithesis – If you’re using agents to code, the problem isn’t writing the code but making sure it didn’t break anything. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. With Antithesis, teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster. Learn more.
• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is growing on me, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.
• Google Cloud Run – a fully managed platform that runs your code directly on top of Google’s scalable infrastructure. Run frontend and backend services, batch jobs, host LLMs, and agents without managing infrastructure. Oh, it has automatic zonal failover out of the box – something that Coinbase could have used a few weeks back! Get started.
In this episode
Navdeep Singh – oftentimes better known as NeetCode – is the creator of NeetCode.io, one of the most popular coding interview preparation platforms and YouTube channels for software engineers. Before building NeetCode full-time, he worked as a software engineer at Amazon and Google.
In this episode of The Pragmatic Engineer, I sit down with Navi to discuss his path from Amazon and Google to building his own startup, why he left Amazon after just two months, what he learned at Google, and the decision to leave a stable engineering career to bet on himself. We also discuss what coding interview preparation teaches beyond passing interviews, the value of going deep on difficult problems, and why systems thinking and domain expertise remain essential engineering skills in the age of AI.
Throughout the conversation, NeetCode makes the case that learning hard things is one of the single best investments an engineer can make, helping build the judgment and expertise that remain valuable no matter how the tools change.
Key observations from Navi
Here are 10 interesting takeaways from our chat:
1. Companies have no real method for evaluating engineers – and likely never did. Navi believes the leetcode-style interview process has persisted because it scales well at large tech companies that need to train hundreds or thousands of interviewers, not because it predicts job performance well.
2. The CAP theorem’s “two-out-of-three” framing is widely taught, but technically shaky.Navi believes this theory of distributed data systems is incomplete, and says he felt validated when researcher and author Martin Kleppmann criticized it. It’s a reminder to think independently and not accept theories without understanding them.
3. Amazon’s intense culture left Navi reluctant to ask questions – which paradoxically, helped at Google. In Navi’s first job, he got used to working alone and not seeking help when needed, and continued this working style at Google. His manager there interpreted that behavior as independence, and as a result, he won rapid promotion from L3 to L4 (mid-level engineering role).
4. The NeetCode YouTube channel took off after he said he’d have to post less. Before viewers knew Navi had got a software engineering job at Google, his audience was small. But it turned out that announcing he’d have to post less for this reason boosted his channel! Suddenly, lots of people wanted to know how he’d landed the role.
5. Cheating tools are helping to resurrect in-person, whiteboard interviews at Google. Navi notes Google has restarted onsite coding interviews because it’s the only way interviewers can be sure that candidates aren’t using AI-powered cheating tools which make data structure and algorithms (DSA) interviews easy to pass.
6. Navi finds AI most valuable as a tech debt and refactoring assistant. He’s using AI to clean up years’ worth of low-quality code on NeetCode’s backend, which also validates the decision to take shortcuts in the knowledge they can be corrected later.
7. ‘Effort’ is becoming the differentiator as AI makes everything else cheap.Navi says how you can prompt almost anything, but the capacity to be engaged with and care about your work, and to defend decisions you make, cannot be prompted by an AI tool. These depend on personal qualities like effort and dedication.
8. Announcements of the death of coding are exaggerated. Despite dramatic improvements in the performance of AI models, Navi does not foresee the majority of engineers being laid off. In fact, he sees the opposite: devs are busier than ever.
9. Humans are likely to remain better at weighing up tradeoffs than LLMs are. It’s a fact that LLMs have become a lot better at coding, but Navi doubts they will be much help in decisions involving judgments about tradeoffs.
10. When hiring for NeetCode, personality traits and motivation matter more than coding skill. Navi’s best recent hire is still an undergrad with little coding experience, but does exceptionally well thanks to possessing high agency. Navi says: “even if they have no idea how to start it, by a week later, they’ll have learned everything about it.”
The Pragmatic Engineer deepdives relevant for this episode
Scheduling note: there will be no edition of The Pulse on Thursday as I’m in San Francisco for the next week and a half, visiting AI labs and startups, and attending the AI Engineer World Fair from next Monday. For the podcast and Tuesday articles, it’s business as usual.Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.As with most of my talks, it came together in stages, including with some input from full subscribers t
Scheduling note: there will be no edition of The Pulse on Thursday as I’m in San Francisco for the next week and a half, visiting AI labs and startups, and attending the AI Engineer World Fair from next Monday. For the podcast and Tuesday articles, it’s business as usual.
Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.
As with most of my talks, it came together in stages, including with some input from full subscribers to the Pragmatic Engineer, with whom I shared my thinking in advance, in ‘Ideas: slow down to speed up when working with AI agents’. Thank you for all the comments!
As fate would have it, just two days beforehand, social media giant Meta appositely provided a real-world case study for my talk, with its most embarrassing outage of all time: users could simply ask the Meta AI to change the email of any account, and the bot happily complied – even if the account belonged to someone else entirely – including a former US president. It was a timely example to kick off the talk with. Check out the full keynote that’s available to view on YouTube:
In this article, I summarize the key parts of my Craft Conference keynote in detail, and some responses received at the event. Full subscribers also have access to the slides, here, and at the foot of this article.
We cover:
Meta: “AI psychosis” in effect? Meta has been destroying its engineering org, and an obsessive focus on AI seems to be one reason for it. For more on this question, check out this deep dive.
Everything’s changed in six months. From around November last year, things changed with a more capable generation of AI agents like Opus 4.5 and GPT-5.4.
How are tech companies changing how they work? Anthropic, OpenAI, Google, Uber, startups, and traditional companies.
Trends. Individual productivity is up, but team productivity’s flat, tokenmaxxing and tooling adoption, vanishing middle management, CEOs and CTOs back to coding, and more.
Trends across software. Falling software quality, GitHub’s constant reliability woes, AI slop overwhelming devs who care about quality, and more.
Advice for software engineers and engineering leaders. Suggestions to help future-proof a career.
Feedback. “It’s happening here too!” is a common theme, and relief for some that it’s not unique to their own workplace.
1. Meta: “AI psychosis” in effect?
I thought it was a made-up story when I read that Meta had enabled account takeovers via a “zero auth” policy; i.e., simply asking the Meta AI bot was sufficient to change any account’s email address. After all, shipping such a regression would fly in the face of security measures, code reviews, automated testing, and metrics. Plus, the company has dedicated Integrity teams whose mission statement is to ensure something like this never happens… And yet, this bug shipped.
It went undetected by anyone at Meta, and high-profile accounts like that of former US president, Barack Obama, were taken over as a result. Instagram’s dedicated Integrity team seems to have discovered the embarrassing issue via the news.
As mentioned, it was two days before the Craft keynote, so there was enough time to ask around at Instagram and Meta. Engineers at the company there told me this disaster was caused by AI-generated, AI-reviewed code, along with layoffs, and by forced reassignments from Integrity teams and elsewhere onto AI labeling and related duties.
Talking Meta at Craft Conference
The problem at Meta seems to be that leadership is aggressively pushing AI, while withdrawing resources and headcount from areas responsible for security, quality, and reliability. Since last week’s deepdive into what’s been happening behind the scenes was published, I’ve learned further details:
Instagram’s design team suffered a 44% cut in headcount during layoffs
The Developer Documentation and Support team had a full 95% headcount reduction during layoffs
Data labeling at the ADO group goes beyond “just” labeling; there are many AI training tasks to do. But these are repetitive, unless you get really creative.
Based on everything I heard from talking with Meta folks, AI-induced behavior was indeed at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted, were also factors in the beyond-embarrassing incident. As reported in last week’s deepdive:
Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.
AI-generated changes with zero human input, with just an additional AI code review, have been very common in recent months across the codebase. The change that caused this outage looked like one of these
Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization”.
If major changes like data labeling assignments and staff tracking are undone, then perhaps things at Meta could return to normal. But so far, the most being done is that leadership has boosted budgets for snacks, travel, and events. Hardly the change needed to restore morale and the former culture!
The comparison to the Lumon corporation in the hit show, Severance, was duly made:
Meta’s worst-ever outage can be interpreted as a warning about what happens when there’s so much focus on AI that the basic health of a company’s main – money-spinning – products is neglected. Instagram, WhatsApp, and Facebook generate the bulk of revenue for Meta, but the company is reallocating more engineers to training the coding model, and aggressively cutting the headcounts of vital orgs to do so – up to the point of not having oncall coverage for key services, and security teams being too stretched to do their jobs.
Am I missing some insight about why it’s more important to build a state-of-the-art, likely-closed AI model that’s good at coding, than it is to keep operating revenue-generating businesses with stable infra?
2. Everything’s changed in six months
Independent, experienced software engineers with zero affiliation to AI labs have been saying for a few months that how we do software engineering has been transformed.
David Heinemeier Hansson (DHH), creator of Ruby on Rails in January:
”Just [in] summer 2025, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”
Simon Willison, creator of Django, in May pinpointed the start of the change to late last year:
“The models released in November 2025 elevated agents to being genuinely useful. We’ve had six months to get used to that idea now; it’s no wonder companies are beginning to spend real money on this technology.”
Teams using agents now ship 5x as many pull requests as two years ago. Here’s data from Linear:
Comparing numbers of pull requests for teams that use AI agents with Linear, vs those that don’t. Source: Linear
Devs using AI harnesses are producing 2.5x as much code versus 18 months ago. Data from Cursor shows that their users, on average, went from adding 3,500 lines of code in January 2025 to 8,600 today:
The size of pull requests is up 3x versus 18 months ago. Also from Cursor:
Line goes up: more lines per PR than ever, today. Source: Cursor
More AI changes are accepted without human review. Data from Cursor shows a big jump in changes being accepted without human review from around February this year, when Opus 4.7 and GPT 5.5 launched:
Less human input than ever, as outlined at the Craft Conference. Source: Cursor
We’re seeing a lot more code generated, and less of it than ever being reviewed by devs. In the relatively short time since AI agents became really good last November, there are more pull requests generated by devs, those pull requests are getting better, and code reviews are harder to keep up with. And so, reviews are less stringent and more changes are shipped to production sans human review! As per my discussions with Meta engineers, these kinds of AI-generated, AI-reviewed pull requests [at Meta, they’re called diffs] are what caused the most recent, embarrassing outage at Instagram.
He personally runs ~5x agents parallel, and ships 20–30 PRs/day
Product requirement documents (PRDs) are dead & prototypes have replaced them inside Anthropic
~100% of Claude Code was generated by Claude in March
~70-90% of code inside Anthropic was generated by Claude
Claude Cowork – another billion-dollar product in terms of revenue potential – was built in just 10 days
Since then, Boris has shared that his workflow has changed to setting up loops to run agents.
OpenAI: moving much faster with AI agents. OpenAI’s Codex team was on the main stage at The Pragmatic Summit in February. Tibo Sottiaux (head of engineering, Codex, OpenAI) shared interesting details on how software development is done in the Codex team:
There’s a “fix this” button integrated into the internal OpenAI mobile app. It makes one-shot fixes to bug reports, which devs review and can merge
AI code review for all code changes. With a tiered approach, some changes can be merged with just AI review, and more important ones need an extra human review
Most devs run several agents in parallel, often walking around with their laptop lids open, so the machine doesn’t enter sleep mode and suspend agents
Code isn’t really written by hand anymore on the Codex team, and is also less common on other teams too
“Taste” is becoming a core skill for working at the company
Codex improves itself: it runs its own test suite, runs improvement tasks overnight, and during team meetings it takes actions on topics discussed
Google: AI widespread. Gemini is not as capable at coding as Claude or Codex, as acknowledged by Google’s CEO, but it’s widely used companywide. The less capable coding model could be hurting AI adoption compared to other companies.
Uber’s Agent Builder: a no-code experience to build agents
The AIFX command line interface:
Uber’s AIFX command line tool
Minion: background agents
Uber’s Minion system: web interface’s appearance
Code Inbox:
Uber’s Code Inbox
Smart Assignments as a neat feature of Code Inbox:
Smart assignment settings for Code Inbox
Risk Profiles: another smart feature inside Code Inbox:
Code Inbox estimates the riskiness of a code change, and brings attention to it
uReview, Uber’s AI code review tool:
AI’s comments can be rated by usefulness
Autocover and Shepherd for large-scale migrations:
Shepherd generates a pull request using a Minion AI agent. Part 2 of the diff (pull request) generated, with code changes
Uber is a good case for learning how much of internal developer infra needs to be rebuilt in order to work well with AI agents. Uber built all the tools above because they needed new, better ways to integrate AI agents into the developer workflow, but couldn’t find anything that worked up to requirements. I’d also point out how much time and effort Uber invested in making code review more efficient. Devs are, indeed, getting overloaded with AI code reviews and Uber’s Code Inbox tries to separate the important pieces of code to review from unimportant ones.
Startups are jumping into using AI agents, although their integrations are more basic. In preparation for the keynote, I talked with several startups about their AI usage. Harnesses like Claude Code, Codex, Cursor, OpenCode and others are popular, and I also noticed most startups are heavily integrating AI agents into Slack, so devs can kick off bugfixes or small feature requests straight from the chat tool.
I observed startups being the most likely to experiment with new AI dev tools; from code review, all the way to AI incident management tools.
“Traditional” companies are also heavily investing in AI dev tools. At the recent Pragmatic Summit in San Francisco, Laura Tacho shared interesting details:
In February, 18,000 Cisco developers used Codex for complex migrations, code review, and refactoring. This was very early – Codex was just starting to gain industry-wide adoption!
JP Morgan Chase built a multi-agent framework for annotation, using multiple specialized agents to label customer interaction data, and judge agents to aggregate and rank results. These are pretty advanced use cases!
In general, “traditional” companies do not seem to be lagging behind in using, paying for, and adopting AI agents and AI developer tools.
4. Industry trends
There are trends I’ve observed around the adoption of AI dev tools:
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.Today, we cover:Meta follow-up. A follow-up on Tuesday’s deepdive into Meta’s ongoing demolition of its own engineering org. Non-engineering teams lost more than 10% of their staff, while Integrity teams were already stretched before the cuts and reallocation.Big implications of the US banning Anthropic’s new model, Fable. The US governmen
The Pulse is a series covering events, insights, and trends within Big Tech and startups. Notice an interesting event or trend? Hit reply and share it with me.
Today, we cover:
Meta follow-up. A follow-up on Tuesday’s deepdive into Meta’s ongoing demolition of its own engineering org. Non-engineering teams lost more than 10% of their staff, while Integrity teams were already stretched before the cuts and reallocation.
Big implications of the US banning Anthropic’s new model, Fable. The US government wants only US citizens to be able to use Fable. That could make China more influential as other countries and non-US companies look to capable open models – most of which come from China.
SpaceX, Cursor, Continue. Elon Musk became a trillionaire with SpaceX’s IPO, while SpaceX acquired Cursor, and Cursor acquired Continue. SpaceX looks like it wants to go head-on against Anthropic and OpenAI.
Industry Pulse. Gemini lead quits for OpenAI, Epic Games’ new open source version control system, Microsoft considers dropping OpenAI for DeepSeek, research confirms LLMs amplify existing expertise, and more.
1. Meta follow-up
Tuesday’s article about Meta destroying its engineering culture made quite the splash in terms of comments and debate online. From inside Meta, there was positive feedback on the accuracy of the deepdive into the situation, and there were also a few helpful clarifications and new information which I’m happy to publish here:
Stream the latest episodeListen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.Brought to You by• Antithesis – if you're using agents to code, the problem isn't writing the code but making sure it didn't break anything. Antithesis goes beyond code review and runs your whole system in faster than real-time and identifies hard to find bugs before your users hit them in production. Antithesis enables tea
Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to You by
• Antithesis – if you're using agents to code, the problem isn't writing the code but making sure it didn't break anything. Antithesis goes beyond code review and runs your whole system in faster than real-time and identifies hard to find bugs before your users hit them in production. Antithesis enables teams like Jane Street, Fly.io, and the etcd community to use agents safely and ship better code, faster. Learn more
• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.
• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. The teams building the smartest AI products out there — Cursor, Notion, Cognition, Anthropic — they all run on turbopuffer.
In this episode
Robert Erez is a principal engineer at Octopus Deploy, and a longtime expert in CI/CD, deployment systems, and software delivery. Rob and I were also once colleagues on the Skype web team, working on large-scale deployments and release processes.
In this episode of ThePragmatic Engineer, I sit down with Rob to discuss how teams deploy software safely and efficiently at scale. We cover Kubernetes, GitOps, platform engineering, progressive delivery, feature flags, cloud development environments, and the growing role of AI in CI/CD workflows. We also get into the tradeoffs in different deployment approaches, why self-hosted software still matters for some organizations, and the recent evolution of software delivery practices.
Key observations on deployments and CI/CD from the conversation with Rob
Here are 10 interesting takeaways from our chat:
1. Roll forward, never backwards. When a system has state – which typically means it uses databases – then doing a rollback can leave the code talking to a schema that’s no longer in sync. Rob’s advice is to not treat a failure in v2 as a trip back to v1, but rather as a push to v3 with the fix in it.
2. GitOps isn’t actually about Git. None of the four pillars of GitOps – 1) declarative, 2) versioned and immutable, 3) pulled, not pushed, 4) continuously reconciled – require Git, although Git can work under these constraints. Yet, the term ‘GitOps’ has made the industry dogmatic about cramming everything into a repo – even things like secrets that absolutely shouldn’t be there!
3. Continuous deployment can be overkill; continuous delivery is more practical. Shipping every single change to prod (continuous deployment) is not as necessary as many people think, Rob says, and there’s often more value in continuous delivery, where changes flow through testing and the deployment process itself is validated. With continuous delivery, you can decide whether to push to production automatically, or click a button once a week.
4. Feature toggles are a better safety net than rollbacks. When something breaks in production, reaching for a toggle to switch a feature off enables you to “stop the bleeding” and then calmly diagnose an issue. Rolling back a feature flag is less nerve-jangling than scrambling to force a redeployment in the middle of the night!
5. One problem with feature flags is that they’re addictive. On the other hand, the ease with which feature flags are added can create a hygiene crisis if they’re continuously added, but not removed. Treat feature-toggle cleanups like a form of gardening and “weed” rolled-out toggles from the codebase.
6. A Git repo can be a bottleneck at scale. Rob mentions that some companies run thousands of independent Kubernetes clusters that pull state from a Git repository. But such clusters can get throttled by the repo, forcing them into workarounds. Pull-based GitOps doesn’t scale infinitely for free.
7. A sizable number of major institutions remain on-prem – and this won’t change. Banks, other financial bodies, and governments, demand full control over their hardware, upgrades, and downtime. That’s why Rob expects this segment won’t move to cloud-based SaaS.
8. Platform teams work at larger companies. These teams earn their keep in big organizations with multiple teams and projects because they offer ways of bringing sanity and focus.
9. There’s a trend of ephemeral environments replacing test/staging environments. Companies used to have a few testers fighting over a handful of static test environments, but today, it’s trivial to spin up a full environment, per-feature branch, pre-merge. This is an “ephemeral” environment for evaluating that things work, which is then torn down once something is merged. It helps speed up the feedback process.
10. AI shifts the CI/CD calculus from speed to risk. Today, shaving ten minutes off the CI build-time matters because a long-running build blocks human devs. But this time saving will be insignificant when an AI agent writes most of the code and “babysits” a slow pipeline without context switching. Then, the new priority will be to reduce the risk of an AI agent shipping a bug to production, so it will make much more sense to run extra, more thorough, tests – and also even slower ones.
The Pragmatic Engineer deepdives relevant for this episode
Hi – this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of senior engineers and engineering leaders. Subscribe to get deepdives like this in your inbox, weekly:Subscribe nowMany subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.For two decades, Meta had a unique, high-performance engineering org; ri
Hi – this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of senior engineers and engineering leaders. Subscribe to get deepdives like this in your inbox, weekly:
For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering.
But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible.
For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks.
We cover:
Meta’s pre-AI engineering culture
Investing in AI and pressing engineers to always use it
Core engineering folks feel treated like trash
Most embarrassing-ever outage
Internal mess
Self-inflicted wounds
Is “AI psychosis” just a Meta issue?
1. Meta’s pre-AI engineering culture
I’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.”
“Move fast and break things”
In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively.
In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964).
At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.
Back then, mantras in Facebook’s little red book were also in print across campus, and included:
Move Fast and Break Things
Done is Better Than Perfect
Fail Harder
What Would You Do If You Weren’t Afraid?
Every Day Feels Like a Week
The Wright Brothers Did Not Have Pilot Licenses
The Foolish Wait
Fortune Favors the Bold
There was genuine focus on building good products. Also from the book:
More from Facebook’s Little Red Book
“Move fast with stable infra” culture
In 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:
“The culture is incredibly engineering-centric: much more than most of Big Tech. This might come from Mark Zuckerberg being an engineer himself, or because much of the innovation in the early days of Facebook came from engineers.
Focus on individual impact. Impact has been the bread and butter of the focus at Facebook. This is very true since the early days, and the focus on generating impact remains.
One detail in common with most Big Tech firms is that both the engineering culture and general culture focus so much on individual impact. This results in some people focusing on short-term, measurable wins and assuming that teamwork and split wins between groups might be less rewarded.
The lack of rigid processes. Facebook seems to have the least amount of processes or standardization across all of Big Tech. Don’t even try to compare it to Amazon’s engineering culture and the countless formal processes there. But even compared to companies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the engineering-centric nature of the company and engineers disliking processes.
Surprisingly little emphasis on testing, documentation or code comments. You’ll find shockingly little automated testing and documentation at Facebook, compared to the rest of Big Tech. Inline code comments are also very rare.
A founder-engineer driven company. Facebook is one of the few Big Tech firms whose founder is an engineer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a software engineer before starting the company. Amazon was the other example of this until recently, but it’s not the case at Google or Apple. There are good examples of smaller companies like Cloudflare, but they’re all younger than Facebook.
Bootcamp. A unique onboarding process, unlike what any other Big Tech firms offer. We cover this more in the Bootcamp & onboarding section.”
Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center.
2. Investing in AI and pressing engineers to always use it
Meta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.
Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s.
This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably.
When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team) as well as a GenAI product organization and released a series of open-weight AI models:
Llama 1: released in Feb 2023, three months after ChatGPT, built by FAIR
Llama 2: in June 2023, built by the GenAI product organization (as well as all subsequent Llama models)
Llama 3: in April 2024. This model was Meta’s most competitive LLM of all, and gained momentum in adoption across the industry
In June that year, Meta acquired a 49% stake in Scale AI to reboot its AI efforts for a whopping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed.
Based on the investment made into Scale AI and Wang, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver.
Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:
Training data and labeling: Scale started, and is still best known, as a provider of high-quality labeled datasets for machine learning and AI training, including code, text, image, video, etc.
RLHF and fine-tuning: A RLHF (reinforcement learning from human feedback) flow which Scale runs, where people give feedback for foundation models, as a “human in the loop” data engine that many leading AI labs use to create better LLMs.
Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it.
Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out.
Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through.
This month, Reuters reported that people’s concerns there are finally being heard:
“Meta is dialing back elements of its plan to collect employee mouse movements, keystrokes and other actions for use as AI training data, it said in an internal memo on Tuesday, following weeks of angry pushback from staffers.
New controls will allow employees to pause the data collection for up to 30 minutes at a time and request exemptions from the initiative, according to the memo, authored by Stephane Kasriel, a vice president in Meta’s AI model-building Superintelligence Labs unit.”
From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation.
Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more.Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation).
“Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:
Engineers were not hired for a specific team (save for at the Staff+, levels, in some cases). They were hired to the company
During a 6-week bootcamp, new hires got familiar with Meta’s engineering culture and chose a team
Team matching meant talking with multiple teams who had headcount, doing small work with them, and finding a match
Internal transfers were easy, and often initiated by engineers
Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects.
“Data labeling” is more involved work, even though a bit repetitive. There are labeling tasks, where you create a website, then look at it and decide if it looks good or not, and give this feedback: this is the “typical” data labeling. But this is the less frequent type of work. But then there are more involved AI training tasks, which looks like this:
Come up with a task that the AI should do
Then write the tests that confirm the result
Package all of this up into a Docker container, using Harbor framework
Then read the code that the AI writes - often doing this based on feedback from several models, and give it feedback
This work is not easy to do — and you can see why you need good software engineers to do it! — but it gets repetitive really quickly. Most engineers who I talked to said that they found it hard to do this work motivated day-in, day-out. Then again, I did talk with an engineer inside the org who said that by varying the technologies they use, and challenging themselves, they found this work motivating and interesting! This engineer expects there to be more software engineering work coming up, after the phase of training passes. There’s a lesson here: when life gives you lemons, you can either complain about it, or you can make lemonade, like this dev has done so.
This training work is secretive across the AI industry, and vendors are offering $100+ per hour for devs to do this type of work. There are rumors across the industry that OpenAI and Anthropic spends more on these training environments (to make the models great at coding) than the model training run itself!
Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left.
One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences.
Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time.
As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.”
I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job.
3. Core engineering folks feel treated like trash
Problem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon.
Forced reassignments to data labeling started to happen. As I covered at the time:
“Understandably, there are mixed feelings about this redeployment [to data labeling], with layoffs coming soon. On Wednesday, 20 May, Meta will announce layoffs. Perhaps those moved to do data labeling could actually be “safer” than colleagues on product teams. Of course, this is speculation, but it would be cruel if Meta cut devs reassigned to data labeling.”
Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.)
Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.
After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations.
Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed.
So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:
“As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.”
The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:
Tracking the keyboards and mouse clicks of all engineers, where legally possible
Reassign a good chunk of engineers to fulltime data labeling
Let staff know that 10% of them will be laid off
Have a culture where devs optimize for any and all metrics measured during PSC
Measure token usage as part of PSC
Shake this mix up well, and what do you get? Two things:
Everyone overuses AI to boost their personal stats. An engineering workforce that pretends to work with as much AI, and as little human input, as possible. It’s a strange incentive where an outage caused by a failure to review code properly is not grounds for dismissal, but writing code by hand – instead of having an AI agent write it – could cost you your job
Every longer-tenured engineer is seeking a new job, or at least considering it. Those who have been around at Meta longer term have seen enough. Let me describe this visually:
Why pretty much every engineer at Meta is looking for a way out, visualised
Fresh data seems to confirm that starting in May, a lot more engineers at Meta are looking for an “out.” Here’s how signups to leading Big Tech interview preparation service interviewing.io from Meta have changed over the last year and a half: we can see a massive jump in May vs the year before, as shared by founder and CEO Aliner Lerner with us:
Signups to interviewing.io’s interview prep service from Meta. Source: interviewing.io.
To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things.
4. Most embarrassing-ever outage
Meta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security.
Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):
“Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.
Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.
The Takeover Flow:
Step 01: Faking the Location & Initiating Support. All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.
Step 02: That’s It. Really, that’s it.
The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.”
This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media!
From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:
Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.
AI-generated changes that saw no human input, just another AI code review, were very common during the last two months, across the codebase. The change that caused this outage looked like one of these
Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization.
Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure.
Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling.
As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage.
5. Internal mess
Wired shares more details on just how bad the situation is inside Meta, right now:
“Someone interrupted a livestreamed, employee-only presentation at Meta earlier this week with an expletive-filled outburst about “being the company’s bitch,” according to a recording heard by WIRED. The individual then asked the people leading the call to write to a specific Meta AI executive and “tell him that he’s a piece of shit.”
The incident, which took place on a call open to thousands of employees, reflects growing frustration inside the company’s Applied AI team, which was formed in March to support the work of AI researchers at Meta Superintelligence Labs. Three current employees tell WIRED there is widespread dissatisfaction with how Meta assembled the unit of about 6,500 engineers and product managers and the drudgework they allege they have been assigned to improve AI models.
“It’s literally the gulag,” one of the employees claims. “You have zero purpose in life all of a sudden, you barely interact with anyone, you just have these tasks every week.”
There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:
During a meeting this week open to all employees at Instagram, Meta chief product officer Chris Cox addressed the “difficult” and “brutal” environment created by the “insanity of this company” in the past few months, according to a recording heard by WIRED. Cox applauded Instagram employees for launching features and serving around 2 billion users amid what he compared to “running a marathon in the middle of a hailstorm and then, like, your teammate gets replaced and then we’re recording you.”
“It’s like what the fuck,” he said, drawing laughs, before repeating himself. “It is like what the fuck.”
6. Self-inflicted wounds
So, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him.
But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:
Mandatory keystroke and mouse tracking to generate training data
Forced data labeling with 4,500+ engineers is to generate high-quality RLHF, surely for Meta’s under-construction coding LLM
Taking away the best engineers from the heart of the business is surely signed off by Mark Zuckerberg, believing that it is more important for Meta to train a coding AI than it is to operate its core business like Instagram, Facebook or Messenger reliably. Oh, did I mention that on Saturday (12 June) Facebook and Instagram had another SEV0, that is, a full-on outage?
Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important.
Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future.
To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation!
Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.
Nice while it lasted
Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave.
7. Is “AI psychosis” just a Meta issue?
It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception?
Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)
“I strongly believe there are entire companies right now under heavy “AI psychosis” and it’s impossible to have rational conversations about it with them. I can’t name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.
I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now it’s... the whole software development industry (maybe the whole world, really).
It’s frightening, because ‘psychosis folks’ operate under an almost absolute “MTTR is all you need” mentality: “it’s fine to ship bugs because the agents will fix them so quickly and at a scale humans can’t do!” We learned in infrastructure that MTTR is great but you can’t yeet resilient systems entirely.
The main issue is I don’t even know how to bring this up to people I know personally, because bringing this topic up leads to immediate dismissals like “no no, it has full test coverage”, or “bug reports are going down” or something, which just don’t paint the whole picture.
We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happen so fast that nobody notices the underlying architecture decaying.
I worry.”
The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly.
Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production.
Takeaways
Most of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company.In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected.
Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to.
And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless.
If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context.
If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta!
It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry.
Connexion
