{"id":500553,"date":"2026-10-27T18:14:21","date_gmt":"2026-10-27T15:14:21","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=500553"},"modified":"2026-10-27T18:14:21","modified_gmt":"2026-10-27T15:14:21","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=500553","title":{"rendered":"The Bitter Lesson Ate Your AI Factory"},"content":{"rendered":"<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>For every AI workflow we define, the next model makes it useless.<\/p>\n<p>What\u2019s worse: most of us keep clinging to the outdated workflows for months after they\u2019ve outlived their usefulness.<\/p>\n<p>Remember how everyone was excited about AI code completion? How about prompt engineering? Harness engineering? Spec-driven development? Loop engineering? Software factory? Dark factory? They are all dead because of the <a href=\"http:\/\/www.incompleteideas.net\/IncIdeas\/BitterLesson.html\" rel=\"noopener nofollow\">bitter lesson<\/a>. To summarize it: no matter what \u201cdomain knowledge\u201d, \u201charness\u201d, or \u201cworkflow\u201d you try to integrate above your AI model, it will be outshone by just a larger, smarter model. This happened in chess when the program that beat Kasparov in 1997 won with massive, deep search, not with the human understanding of chess that most researchers had spent years encoding. And it keeps happening to us today.<\/p>\n<p>That\u2019s why whenever somebody tries to build yet another factory or a really constraining skill, I roll my eyes since modern models perform <strong>worse<\/strong> with these tools. There are even benchmark runs showing that: <a href=\"https:\/\/danluu.com\/agentic-testing\/\" rel=\"noopener nofollow\">Dan Luu tried 26 testing instructions and a few popular testing skills on agents implementing Zstd<\/a>, and plain \u201cno additional instructions\u201d did well above average while the popular skills underperformed; <a href=\"https:\/\/x.com\/kunchenguid\/status\/2064196342248030352\" rel=\"noopener nofollow\">Kun Chen measured that asking agents to do TDD makes them worse<\/a>. On DeepSWE, near-bare mini-swe-agent did as well as or better than Claude Code and Codex CLI.<\/p>\n<p>Every time I discuss outdated and over-engineered AI workflows, I find it useful to remember what problem the workflow is used to solve and what model made it unnecessary:<\/p>\n<figure class=\"\"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/extt\/af\/f5\/dd\/aff5dd691904fe85d64ab7e8c9feb331.png\" alt=\"Every workflow dies to the next model\" sizes=\"(max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/extt\/af\/f5\/dd\/aff5dd691904fe85d64ab7e8c9feb331.png 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/extt\/af\/f5\/dd\/aff5dd691904fe85d64ab7e8c9feb331.png 781w\" loading=\"lazy\" decode=\"async\"\/><\/p>\n<div><figcaption>Every workflow dies to the next model<\/figcaption><\/div>\n<\/figure>\n<ul>\n<li>\n<p>OpenAI o1 made \u201cfew-shot\u201d, \u201cchain-of-thought\u201d, etc. prompting irrelevant because of reasoning<\/p>\n<\/li>\n<li>\n<p>Sonnet 3.5 gave birth to vibe coding<\/p>\n<\/li>\n<li>\n<p>Sonnet 3.7 made meticulously crafted prompts irrelevant and moved many people from tab-complete to generating whole functions and modules<\/p>\n<\/li>\n<li>\n<p>Opus 4 with Claude Code made real agentic coding viable for weekend projects; Opus 4.5 made it viable for work. At that point many people from my bubble started writing 70-80% of code with it, which finished off any hope for AI tab-complete and \u201ccompact-your-codebase-into-one-prompt\u201d tools<\/p>\n<\/li>\n<li>\n<p>GPT-5.4 kept track of a long-running list of tasks without the help of a plan mode, spec-driven development or a Ralph loop, making all three unnecessary. The only part of spec-driven that\u2019s still useful is getting requirements from a user, which is better accomplished by the <code>\/grill-with-docs<\/code> skill<\/p>\n<\/li>\n<li>\n<p>Fable 5 caught up in terms of not losing track of work and showed us that sufficiently smart models could make good decisions on their own, lowering our need for a variety of guardrails and checks. But it was still too expensive for most people to use as a driver, so \u201creviewer\u201d workflows got popular: a cheaper model does the work and asks Fable for decisions<\/p>\n<\/li>\n<li>\n<p>GPT-5.6 Sol along with OpenAI\u2019s outstanding computer\/browser use made it possible to delegate much larger and human-y tasks to agents, making MCPs optional for most websites and apps. I have almost forgotten what dashboards for all my infra look like because these days agents are more capable than me at taking actions there<\/p>\n<\/li>\n<li>\n<p>Opus 5.5 fixed the prose issue of the Claude family and could make good decisions like Fable at a fraction of the cost, removing the need for the \u201creviewer\u201d workflow and for factories when using frontier models (though arguably factories were not necessary even at GPT 5.6 Sol stage)<\/p>\n<\/li>\n<\/ul>\n<p>Notice the orange curves: the workflows either became popular or kept being popular <strong>after<\/strong> they got (mostly) useless for frontier models. And the time period between the introduction of the workflow and its death keeps shrinking.<\/p>\n<p>These days a naive and concise prompt into Claude Code with Opus 5.5 will perform significantly better than every AI factory because factories needlessly constrain Opus. A default Opus 5.5 can make much better decisions by <a href=\"https:\/\/en.wikipedia.org\/wiki\/Shuhari#Definition\" rel=\"noopener nofollow\">going beyond the constraints and rules<\/a> that factories commonly put on it since the written rules cannot foresee every situation and invariant.<\/p>\n<p>The rate of change is so fast that most AI courses and books are outdated by the time they come out. So I advise all readers to instead focus on the short articles the labs publish when they release new models. Except for Gemini, of course: they are too far from the frontier to matter in this discussion.<\/p>\n<p>Whenever you are trying out or building yet-another-really-modern-AI-workflow, think about the bitter lesson: your workflow is likely already outdated.<\/p>\n<details class=\"spoiler\">\n<summary>What about local models?<\/summary>\n<div class=\"spoiler__content\">\n<p>I am fully aware that my logic only applies to people using frontier models. Of course, many of these workflows are still valid in the Qwen world, especially if you are aiming at using 27b-sized LLMs. My concern is strictly with the people trying to apply said workflows to frontier, which is still a lot of people.<\/p>\n<\/div>\n<\/details>\n<p>Originally published at <a href=\"https:\/\/zmievsa.org\/posts\/the-bitter-lesson-ate-your-ai-factory\/\" rel=\"noopener nofollow\">https:\/\/zmievsa.org\/posts\/the-bitter-lesson-ate-your-ai-factory\/<\/a><\/p>\n<\/div>\n<p>\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/1100198\/\">https:\/\/habr.com\/ru\/articles\/1100198\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>For every AI workflow we define, the next model makes it useless.What\u2019s worse: most of us keep clinging to the outdated workflows for months after they\u2019ve outlived their usefulness.Remember how everyone was excited about AI code completion? How about prompt engineering? Harness engineering? Spec-driven development? Loop engineering? Software factory? Dark factory? They are all dead because of the bitter lesson. To summarize it: no matter what \u201cdomain knowledge\u201d, \u201charness\u201d, or \u201cworkflow\u201d you try to integrate above your AI model, it will be outshone by just a larger, smarter model. This happened in chess when the program that beat Kasparov in 1997 won with massive, deep search, not with the human understanding of chess that most researchers had spent years encoding. And it keeps happening to us today.That\u2019s why whenever somebody tries to build yet another factory or a really constraining skill, I roll my eyes since modern models perform worse with these tools. There are even benchmark runs showing that: Dan Luu tried 26 testing instructions and a few popular testing skills on agents implementing Zstd, and plain \u201cno additional instructions\u201d did well above average while the popular skills underperformed; Kun Chen measured that asking agents to do TDD makes them worse. On DeepSWE, near-bare mini-swe-agent did as well as or better than Claude Code and Codex CLI.Every time I discuss outdated and over-engineered AI workflows, I find it useful to remember what problem the workflow is used to solve and what model made it unnecessary:Every workflow dies to the next modelOpenAI o1 made \u201cfew-shot\u201d, \u201cchain-of-thought\u201d, etc. prompting irrelevant because of reasoningSonnet 3.5 gave birth to vibe codingSonnet 3.7 made meticulously crafted prompts irrelevant and moved many people from tab-complete to generating whole functions and modulesOpus 4 with Claude Code made real agentic coding viable for weekend projects; Opus 4.5 made it viable for work. At that point many people from my bubble started writing 70-80% of code with it, which finished off any hope for AI tab-complete and \u201ccompact-your-codebase-into-one-prompt\u201d toolsGPT-5.4 kept track of a long-running list of tasks without the help of a plan mode, spec-driven development or a Ralph loop, making all three unnecessary. The only part of spec-driven that\u2019s still useful is getting requirements from a user, which is better accomplished by the \/grill-with-docs skillFable 5 caught up in terms of not losing track of work and showed us that sufficiently smart models could make good decisions on their own, lowering our need for a variety of guardrails and checks. But it was still too expensive for most people to use as a driver, so \u201creviewer\u201d workflows got popular: a cheaper model does the work and asks Fable for decisionsGPT-5.6 Sol along with OpenAI\u2019s outstanding computer\/browser use made it possible to delegate much larger and human-y tasks to agents, making MCPs optional for most websites and apps. I have almost forgotten what dashboards for all my infra look like because these days agents are more capable than me at taking actions thereOpus 5.5 fixed the prose issue of the Claude family and could make good decisions like Fable at a fraction of the cost, removing the need for the \u201creviewer\u201d workflow and for factories when using frontier models (though arguably factories were not necessary even at GPT 5.6 Sol stage)Notice the orange curves: the workflows either became popular or kept being popular after they got (mostly) useless for frontier models. And the time period between the introduction of the workflow and its death keeps shrinking.These days a naive and concise prompt into Claude Code with Opus 5.5 will perform significantly better than every AI factory because factories needlessly constrain Opus. A default Opus 5.5 can make much better decisions by going beyond the constraints and rules that factories commonly put on it since the written rules cannot foresee every situation and invariant.The rate of change is so fast that most AI courses and books are outdated by the time they come out. So I advise all readers to instead focus on the short articles the labs publish when they release new models. Except for Gemini, of course: they are too far from the frontier to matter in this discussion.Whenever you are trying out or building yet-another-really-modern-AI-workflow, think about the bitter lesson: your workflow is likely already outdated.What about local models?I am fully aware that my logic only applies to people using frontier models. Of course, many of these workflows are still valid in the Qwen world, especially if you are aiming at using 27b-sized LLMs. My concern is strictly with the people trying to apply said workflows to frontier, which is still a lot of people.Originally published at https:\/\/zmievsa.org\/posts\/the-bitter-lesson-ate-your-ai-factory\/\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1100198\/<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-500553","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/500553","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=500553"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/500553\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=500553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=500553"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=500553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}