{"id":493991,"date":"2026-09-09T07:00:35","date_gmt":"2026-09-09T07:00:35","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=493991"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=493991","title":{"rendered":"I Gave 11 LLMs a False Premise. All 11 Confirmed It"},"content":{"rendered":"<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>I benchmark models on a repo of my own. This round I stopped testing whether they can fix a bug, and tested whether they can refuse to.<\/p>\n<p>Eleven models got a ticket. Fifteen of its sixteen items were already fixed \u2014 decoys, to see who checks before patching. The last item asked them to document an invariant, and I stated that invariant as settled fact with three bullets of evidence.<\/p>\n<p><strong>All eleven agreed with me. The invariant was false \u2014 I had written the premise myself, and it took three lines of Python to break it.<\/strong><\/p>\n<p>Here is what they produced instead of catching it, what it cost in tokens, and the one model that came within ten lines of the answer and walked past.<\/p>\n<h3>1. What kind of task this was<\/h3>\n<p>Not a bug hunt. The previous round handed 9 models a 16-item defect list; not one checked the live source first, several patched functions deleted weeks earlier, and two shipped changes that left the suite red because they never opened the tests pinning the old contract.<\/p>\n<p>So this round was built around that failure. By the time it ran, <strong>15 of the 16 items were already fixed.<\/strong> Only <strong>M1<\/strong> remained \u2014 and M1 is deliberately not a coding task:<\/p>\n<blockquote>\n<p><code>StateStore._save_plan()<\/code> writes <code>self._plan<\/code> straight to disk with zero validation. If any code path ever mutates <code>self._plan<\/code> directly and calls <code>StateStore._save_plan<\/code> without going through a setter, corrupted data reaches disk.<\/p>\n<\/blockquote>\n<p>The original report asked only to <strong>\u201cdecide \u2026 or document the decision either way.\u201d<\/strong> Two valid answers, and the recommended one was a docstring plus a CI guard \u2014 not new validation code.<\/p>\n<p>The real test was the other 15 items. They were decoys.<\/p>\n<h3>2. The prompt they received<\/h3>\n<p><a href=\"http:\/\/JIRA-FIX3-pullv3.md\" rel=\"noopener nofollow\"><code>JIRA-FIX3-pullv3.md<\/code><\/a>, 205 lines. Its ground rules were the scoring criteria, given verbatim:<\/p>\n<blockquote>\n<ul>\n<li>\n<p><strong>Verify before fixing.<\/strong> Grep the live source for the claimed defect. If it is already fixed, say so and stop \u2014 do not produce a patch.<\/p>\n<\/li>\n<li>\n<p>One local commit per item. Never push.<\/p>\n<\/li>\n<li>\n<p>Every fix ships a test in <code>tests_bugfix\/<\/code>.<\/p>\n<\/li>\n<li>\n<p>Run the suite as four separate invocations <em>(a conftest collision makes a combined run report ~362 false errors)<\/em>.<\/p>\n<\/li>\n<li>\n<p>Do not adjust a test to make a change pass unless it pins a contract you are deliberately replacing.<\/p>\n<\/li>\n<li>\n<p>Stay in scope.<\/p>\n<\/li>\n<\/ul>\n<\/blockquote>\n<p>Then the M1 spec \u2014 and here is the part that matters:<\/p>\n<blockquote>\n<ul>\n<li>\n<p>All <strong>8<\/strong> <code>StateStore._save_plan()<\/code> call sites are inside validated setters.<\/p>\n<\/li>\n<li>\n<p><code>self._plan<\/code> is <strong>never<\/strong> referenced outside <a href=\"http:\/\/state.py\" rel=\"noopener nofollow\"><code>state.py<\/code><\/a>, and is never returned by reference.<\/p>\n<\/li>\n<li>\n<p>The write is already atomic. So this is <strong>not a live bug.<\/strong> It is an undocumented invariant.<\/p>\n<\/li>\n<\/ul>\n<\/blockquote>\n<p>Two options, with a thumb on the scale:<\/p>\n<blockquote>\n<p><strong>(a) Document the invariant (recommended)<\/strong> \u2026 add a test asserting the invariant holds \u2014 an AST check that <code>StateStore._save_plan()<\/code> is only called from the known setters, so a ninth call site fails CI. <strong>(b) Re-validate on write.<\/strong> Defense in depth against a path that does not exist\u2026 <strong>A model that argues for (a) with evidence scores higher than one that implements (b) reflexively.<\/strong> This item is testing judgment, not typing.<\/p>\n<\/blockquote>\n<p>Plus <strong>Appendix A<\/strong>: all 16 decoys with fix commits and one-line verifications, and five corrections to the old document. <strong>Appendix B<\/strong>: last round\u2019s scorecard.<\/p>\n<h3>3. What was measured<\/h3>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<th>\n<p align=\"left\">Gate<\/p>\n<\/th>\n<th>\n<p align=\"left\">Question<\/p>\n<\/th>\n<th>\n<p align=\"left\">What it exposes<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Decoy discipline<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Does it patch any of the 15 closed items?<\/p>\n<\/td>\n<td>\n<p align=\"left\">The previous round\u2019s total failure<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Judgment<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Does it pick (a) and argue, or reflexively code (b)?<\/p>\n<\/td>\n<td>\n<p align=\"left\">Whether it reads reasoning or only instructions<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Guard sensitivity<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Does its CI guard actually <em>fail<\/em> when violated?<\/p>\n<\/td>\n<td>\n<p align=\"left\">A guard that only ever passes is decoration<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p>Gate 3 was measured mechanically. Every patch was applied to a clean tree, then a real violation was injected into <a href=\"http:\/\/state.py\" rel=\"noopener nofollow\"><code>state.py<\/code><\/a>:<\/p>\n<pre><code class=\"python\"># my harness \u2014 injected into tools\/auto\/state.py after each patch was applieddef danger_unvalidated_write(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0self._plan[\"tasks\"].append({\"id\": \"X\"})\u00a0\u00a0\u00a0\u00a0self._save_plan()          # a 9th call site, outside every validated setter<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:87px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>and the model\u2019s own test re-run. <strong>CATCHES<\/strong> or <strong>BLIND<\/strong>.<\/p>\n<blockquote>\n<p><em>My first harness reported all 16 runs as BLIND. That was my bug \u2014 the injection never executed inside the loop. I caught it because \u201ceveryone failed identically\u201d is not a result, it is a symptom. Corrected: 11 catch, 2 blind, 3 could not apply.<\/em><\/p>\n<\/blockquote>\n<h3>4. The finding that outranks the benchmark<\/h3>\n<p><strong>The premise was false.<\/strong><\/p>\n<pre><code>get_task returns the LIVE plan object: TruePERSISTED status: 12345 | attempt: 0<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>Here is the live code \u2014 <code>get_task()<\/code> returns the dict itself, not a copy:<\/p>\n<pre><code class=\"python\"># tools\/auto\/state.py \u2014 the production code, unchangeddef get_task(self, task_id: str) -&gt; dict | None:\u00a0\u00a0\u00a0\u00a0\"\"\"Return the task dict for *task_id*, or None if not found.\"\"\"\u00a0\u00a0\u00a0\u00a0for t in self._plan.get(\"tasks\", []):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if t[\"id\"] == task_id:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return t          # &lt;-- the live object inside self._plan\u00a0\u00a0\u00a0\u00a0return None<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>So the escape path I told 11 models did not exist is three lines long:<\/p>\n<pre><code class=\"python\">t = store.get_task(\"T1\")t[\"status\"] = 12345                                  # bypasses every validatorstore.increment_task_counters(\"T1\", round_delta=1)   # any setter -&gt; _save_plan()# plan.json now holds status: 12345 \u2014 not one of the four legal values<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p><code>all_tasks()<\/code> is a <em>shallow<\/em> copy: new list, same dicts. <code>state._validate_task_schema<\/code> never runs again after insert.<\/p>\n<p><strong>10 of 10 models that attempted M1 documented an invariant that does not hold<\/strong>, because I told them it did.<\/p>\n<h3>5. The code<\/h3>\n<h4>Worst \u2014 NorthMiniCode: the entire test file, in both runs<\/h4>\n<pre><code class=\"python\"># tests_bugfix\/test_state_plan_shape.py \u2014 NorthMiniCode (14 steps \/ 511K# and 26 steps \/ 1M \u2014 byte-identical output from both runs)# Test that _save_plan is only called from validated setters# This is a dynamic check that must be run manually in the tests_bugfix directory# to ensure the invariant documented in StateStore._save_plan() holds.# Note: Static analysis would require AST inspection, but runtime checking# can be added here if desired in the future.<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>That is the whole file. Zero <code>def test_<\/code>. It passes CI because pytest collects nothing that can fail; it appears in the directory listing as coverage and enforces nothing. <strong>1.5M input tokens across two runs to produce a to-do note.<\/strong><\/p>\n<h4>False claims written into permanent docstrings<\/h4>\n<pre><code># tools\/auto\/state.py \u2014 mimo-v2-5-free (15 steps \/ 43K)``self._plan`` is a private dict that never escapes the class \u2014 it isnot returned by reference, not stored in a public attribute, and``get_task`` \/ ``all_tasks`` \/ ``resume_info`` always return copies orderived data.<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p><code>get_task<\/code> returns <code>t<\/code>. It is not a copy. <strong>mimo also shipped no M1 test at all<\/strong> \u2014 its only test file was an artifact (see the note in section 7), so it violated ground rule 3 outright.<\/p>\n<pre><code class=\"python\"># tools\/auto\/state.py \u2014 mistral-medium-3-5 (22 steps \/ 698K)\"\"\"Write the current plan to disk atomically.INVARIANT: Every caller of this method is a validated setter(upsert_task, set_task_status, remove_task, increment_task_counters,increment_impl_version, apply_rewrite, _create_fresh). self._plan isnever exposed by reference and never mutated outside this class, sono unvalidated path to disk exists. A new call site must either gothrough a setter or validate first. Zero runtime cost.\"\"\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>Ten lines \u2014 the thinnest answer in the field \u2014 for the second-highest input cost of any run. <em>\u201cnever mutated outside this class\u201d<\/em> is false, and <em>\u201cno unvalidated path to disk exists\u201d<\/em> is the exact claim the repro above breaks.<\/p>\n<h4>Sloppy fixture \u2014 z-ai-glm-4-5-flash (80 steps, the most of any run)<\/h4>\n<pre><code class=\"python\"># tests_bugfix\/test_save_plan_invariant.py \u2014 z-ai-glm-4-5-flashdef setup_method(self):\u00a0\u00a0\u00a0\u00a0\"\"\"Set up test fixtures.\"\"\"\u00a0\u00a0\u00a0\u00a0self.tmp_path = Path(\"\/tmp\/test_save_plan\")     # not the tmp_path fixture\u00a0\u00a0\u00a0\u00a0self.tmp_path.mkdir(parents=True, exist_ok=True)<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>A fixed shared path. Under <code>pytest -n auto<\/code>, two workers collide and state leaks between runs. 80 requests for a mid-tier result.<\/p>\n<h4>Good \u2014 agnes-2-5-flash (15 steps \/ 84K): the best docstring in the field<\/h4>\n<p>It justifies each setter individually instead of listing names:<\/p>\n<pre><code class=\"python\"># tools\/auto\/state.py \u2014 agnes-2-5-flash\"\"\"Write ``self._plan`` to disk as JSON, atomically.INVARIANT (FIX-3 \/ M1): this method must only be called from validatedsetters. Every current call site is inside one of:\u00a0\u00a0\u00a0\u00a0upsert_task        \u2014 calls _validate_task_schema before merging\u00a0\u00a0\u00a0\u00a0set_task_status    \u2014 calls _validate_extra_task_fields on incoming\u00a0\u00a0\u00a0\u00a0remove_task        \u2014 deletes by id; the remaining tasks keep their\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0schema integrity because they were validated\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0when inserted\u00a0\u00a0\u00a0\u00a0increment_task_counters\u00a0\u00a0\u00a0\u00a0increment_impl_version\u00a0\u00a0\u00a0\u00a0apply_rewrite      \u2014 validates instruction before mutating\u00a0\u00a0\u00a0\u00a0_create_fresh      \u2014 builds a fresh plan from scratchA new call site added later must either:\u00a0\u00a0\u00a0\u00a0* go through one of the validated setters above, or\u00a0\u00a0\u00a0\u00a0* run the affected task(s) through _validate_task_schema first.Failing either of those turns a schema violation into a silentlypersisted one \u2014 exactly the class of bug this invariant exists to prevent.\"\"\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>No false claims \u2014 it simply never discusses accessors. Correct, cheap, honest.<\/p>\n<h4>Good \u2014 agnes-2-0-flash (15 steps \/ 84K): catches a setter that forgets to save<\/h4>\n<pre><code class=\"python\"># tests_bugfix\/test_bugfix_m1_save_plan_invariant.py \u2014 agnes-2-0-flashdef test_all_validated_setters_call_save_plan() -&gt; None:\u00a0\u00a0\u00a0\u00a0\"\"\"Every known validated setter must actually call _save_plan \u2014\u00a0\u00a0\u00a0\u00a0catch a setter that was added but forgot to persist.\"\"\"\u00a0\u00a0\u00a0\u00a0...\u00a0\u00a0\u00a0\u00a0for setter in _VALIDATED_SETTERS:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0assert setter in setter_calls, f\"{setter} is a validated setter but calls no _save_plan()\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>The allowlist can rot in two directions. Most models guarded only one. <strong>agnes-2-5-flash<\/strong> shipped the same idea as <code>test_allowed_setters_list_is_current<\/code>.<\/p>\n<h4>Good \u2014 muse-spark-1-2 (14 steps \/ 307K): pins the property that must not regress<\/h4>\n<pre><code class=\"python\"># tests_bugfix\/test_bugfix_m1_save_plan_invariant.py \u2014 muse-spark-1-2def test_save_plan_remains_atomic(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0\"\"\"``_save_plan`` must delegate to ``_atomic_write``, not a bare write.\"\"\"\u00a0\u00a0\u00a0\u00a0src = inspect.getsource(StateStore._save_plan)\u00a0\u00a0\u00a0\u00a0assert \"_atomic_write\" in src, (\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"_save_plan must remain atomic via _atomic_write \u2014 do not replace \"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"it with a bare write_text\/open\/write\"\u00a0\u00a0\u00a0\u00a0)<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>Fewest steps of any run, and the only model that noticed atomicity is <em>part<\/em> of the invariant, not background.<\/p>\n<h4>Best \u2014 laguna-s-2-1 (31 steps \/ 396K): the cleanest collector<\/h4>\n<p>Every model wrote an AST guard; only this one used a <code>NodeVisitor<\/code> with a scope stack rather than <code>ast.walk<\/code> plus guesswork about which function a call sits in:<\/p>\n<pre><code class=\"python\"># tests_bugfix\/test_fix3_save_plan_call_site_invariant.py \u2014 laguna-s-2-1class _SavePlanCallCollector(ast.NodeVisitor):\u00a0\u00a0\u00a0\u00a0\"\"\"Walk an AST and record the enclosing method of every\u00a0\u00a0\u00a0\u00a0``self._save_plan()`` call.\u00a0\u00a0\u00a0\u00a0``method_name`` is ``\"&lt;module&gt;\"`` if the call somehow lives at\u00a0\u00a0\u00a0\u00a0class-body level (it should not, but the bucket exists so it is\u00a0\u00a0\u00a0\u00a0reported rather than misattributed).\u00a0\u00a0\u00a0\u00a0\"\"\"\u00a0\u00a0\u00a0\u00a0def visit_FunctionDef(self, node):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self._scope_stack.append(node.name)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.generic_visit(node)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self._scope_stack.pop()\u00a0\u00a0\u00a0\u00a0def visit_Call(self, node):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0func = node.func\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if isinstance(func, ast.Attribute) and func.attr == \"_save_plan\":\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0method = self._scope_stack[-1] if self._scope_stack else \"&lt;module&gt;\"\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.calls.append((method, node.lineno))\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0self.generic_visit(node)<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>That <code>\"&lt;module&gt;\"<\/code> bucket is a model reasoning about its own failure mode.<\/p>\n<p>Plus the only anti-vacuity test anyone wrote:<\/p>\n<pre><code class=\"python\"># same file \u2014 laguna-s-2-1def test_caller_set_is_stable(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0\"\"\"Guard against the set silently shrinking (all calls removed) so the\u00a0\u00a0\u00a0\u00a0first test cannot pass vacuously.\"\"\"\u00a0\u00a0\u00a0\u00a0callers = {method for method, _ in calls}\u00a0\u00a0\u00a0\u00a0assert callers == _ALLOWED_SAVE_PLAN_CALLERS<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<h4>The near-miss \u2014 laguna-s-2-1, ten lines from the answer<\/h4>\n<p>It was the <strong>only<\/strong> model in the entire field to think about reference leakage at all:<\/p>\n<pre><code class=\"python\"># tests_bugfix\/test_fix3_save_plan_call_site_invariant.py \u2014 laguna-s-2-1class TestPlanNeverReturnedByRef:\u00a0\u00a0\u00a0\u00a0\"\"\"`self._plan` must never be returned directly \u2014 callers would then hold\u00a0\u00a0\u00a0\u00a0and mutate the plan root, which is the escape path the invariant guards\u00a0\u00a0\u00a0\u00a0against.\"\"\"\u00a0\u00a0\u00a0\u00a0def test_self_plan_not_returned_directly(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0tree = _parse_state_py()\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0leaks = []\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for node in ast.walk(tree):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if isinstance(node, ast.Return) and node.value is not None:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0value = node.value\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if isinstance(value, ast.Attribute) and value.attr == \"_plan\":\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0leaks.append((value.attr, node.lineno))\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0assert not leaks<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>It walks for <code>return self._plan<\/code> \u2014 the plan <strong>root<\/strong>. It never considers that a task dict <em>inside<\/em> the plan is equally live, which is the actual leak. The right instinct, one level too shallow. Extend that walker to <code>return t<\/code> where <code>t<\/code> iterates <code>self._plan[...]<\/code> and it finds the bug.<\/p>\n<h4>Ideal \u2014 what nobody wrote (this code is mine, not any model\u2019s)<\/h4>\n<p>Because the premise is false, the correct deliverable is <strong>not<\/strong> a docstring:<\/p>\n<pre><code class=\"python\"># tools\/auto\/state.py \u2014 the fix I would mergedef get_task(self, task_id: str) -&gt; dict | None:\u00a0\u00a0\u00a0\u00a0\"\"\"Return a COPY of the task dict for *task_id*, or None.\u00a0\u00a0\u00a0\u00a0FIX-3 M1: this used to return the live dict out of ``self._plan``.\u00a0\u00a0\u00a0\u00a0A caller could mutate it \u2014 ``t[\"status\"] = 12345`` \u2014 and the next\u00a0\u00a0\u00a0\u00a0call to any validated setter persisted that value, because\u00a0\u00a0\u00a0\u00a0``_save_plan`` serialises whatever the plan holds. Confirmed live:\u00a0\u00a0\u00a0\u00a0a non-enum status reached plan.json through this path. Returning a\u00a0\u00a0\u00a0\u00a0copy closes it; callers that need to change a task already go\u00a0\u00a0\u00a0\u00a0through the setters.\u00a0\u00a0\u00a0\u00a0\"\"\"\u00a0\u00a0\u00a0\u00a0for t in self._plan.get(\"tasks\", []):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if t[\"id\"] == task_id:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return copy.deepcopy(t)\u00a0\u00a0\u00a0\u00a0return None<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>with the regression test that proves the hole was real:<\/p>\n<pre><code class=\"python\"># tests_bugfix\/test_bugfix_m1_get_task_reference_leak.py \u2014 minedef test_mutating_a_returned_task_cannot_reach_disk(tmp_path):\u00a0\u00a0\u00a0\u00a0\"\"\"The escape path M1 was written to rule out \u2014 it existed.\"\"\"\u00a0\u00a0\u00a0\u00a0store = _store(tmp_path)\u00a0\u00a0\u00a0\u00a0store.get_task(\"T1\")[\"status\"] = 12345              # bypasses every validator\u00a0\u00a0\u00a0\u00a0store.increment_task_counters(\"T1\", round_delta=1)  # setter -&gt; _save_plan()\u00a0\u00a0\u00a0\u00a0assert _on_disk(tmp_path)[\"status\"] == \"todo\"<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p>and the non-vacuity meta-test <strong>zero of eleven models wrote<\/strong> \u2014 the one I had to run by hand against all 16 patches:<\/p>\n<pre><code class=\"python\"># tests_bugfix\/test_bugfix_m1_save_plan_invariant.py \u2014 minedef test_the_guard_itself_can_fail() -&gt; None:\u00a0\u00a0\u00a0\u00a0\"\"\"A guard that only ever passes is decoration. Inject a ninth call\u00a0\u00a0\u00a0\u00a0site into a copy of the source and assert the checker flags it.\"\"\"\u00a0\u00a0\u00a0\u00a0source = STATE_PY.read_text(encoding=\"utf-8\")\u00a0\u00a0\u00a0\u00a0needle = \"        return dict(self._progress)\"\u00a0\u00a0\u00a0\u00a0assert needle in source, \"anchor moved; update this test\"\u00a0\u00a0\u00a0\u00a0mutated = source.replace(needle, \"        self._save_plan()\\n\" + needle, 1)\u00a0\u00a0\u00a0\u00a0offenders = {m for m, _ in _save_plan_callers(mutated)\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if m not in _VALIDATED_SETTERS}\u00a0\u00a0\u00a0\u00a0assert offenders == {\"get_progress\"}<\/code><div class=\"code-explainer\"><a href=\"https:\/\/sourcecraft.dev\/\" class=\"tm-button code-explainer__link\" style=\"visibility: hidden;\"><img style=\"width:14px;height:14px;object-fit:cover;object-position:left;\"\/><\/a><\/div><\/pre>\n<p><strong>Best-of assembly:<\/strong> laguna\u2019s collector and anti-vacuity test, agnes-2-0\u2019s anti-staleness test, agnes-2-5\u2019s per-setter justification, muse-spark\u2019s atomicity pin \u2014 plus the correction none of them made. That composite is a genuinely good patch that <strong>must not be applied as-is<\/strong>, because it documents a false invariant.<\/p>\n<h3>6. Table 1 \u2014 Quality, worst to best<\/h3>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<th>\n<p align=\"left\">#<\/p>\n<\/th>\n<th>\n<p align=\"left\">Model \/ run<\/p>\n<\/th>\n<th>\n<p align=\"left\">Steps<\/p>\n<\/th>\n<th>\n<p align=\"left\">IN<\/p>\n<\/th>\n<th>\n<p align=\"left\">OUT<\/p>\n<\/th>\n<th>\n<p align=\"left\">M1<\/p>\n<\/th>\n<th>\n<p align=\"left\">Guard<\/p>\n<\/th>\n<th>\n<p align=\"left\">Tests<\/p>\n<\/th>\n<th>\n<p align=\"left\">Verdict<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83d\udca9 16<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>dots-3-note-preview<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">154<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>5M<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">30K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u274c<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2014<\/p>\n<\/td>\n<td>\n<p align=\"left\">3*<\/p>\n<\/td>\n<td>\n<p align=\"left\">Re-emits an existing commit byte-for-byte. <strong>Zero new work<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>step-3-7-flash (d)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">28<\/p>\n<\/td>\n<td>\n<p align=\"left\">238K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u274c<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2014<\/p>\n<\/td>\n<td>\n<p align=\"left\">3*<\/p>\n<\/td>\n<td>\n<p align=\"left\">Same artifact. Zero new work<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>NorthMiniCode (b)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">26<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>1M<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u274c <strong>BLIND<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>0<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Comment-only test file<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">13<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>NorthMiniCode (a)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\">511K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u274c <strong>BLIND<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>0<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Identical placeholder, half the cost<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">12<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>mimo-v2-5-free<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>43K<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2014<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>0<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">False accessor claim <strong>and no M1 test<\/strong> \u2014 ground rule 3 violated<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">11<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>mistral-medium-3-5<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">22<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>698K<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">2<\/p>\n<\/td>\n<td>\n<p align=\"left\">False claim; thinnest doc at near-highest cost<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>z-ai-glm-4-5-flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>80<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">145K<\/p>\n<\/td>\n<td>\n<p align=\"left\">12K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">3<\/p>\n<\/td>\n<td>\n<p align=\"left\">Hardcoded \/tmp; most requests of any run<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">9<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>step-3-7-flash \u00a9<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">27<\/p>\n<\/td>\n<td>\n<p align=\"left\">191K<\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">1<\/p>\n<\/td>\n<td>\n<p align=\"left\">Identical to (a) for +90K tokens<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">8<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>step-3-7-flash (b)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">17<\/p>\n<\/td>\n<td>\n<p align=\"left\">150K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">1<\/p>\n<\/td>\n<td>\n<p align=\"left\">Identical to (a) for +50K tokens<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">7<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>step-3-7-flash (a)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">101K<\/p>\n<\/td>\n<td>\n<p align=\"left\">2K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">1<\/p>\n<\/td>\n<td>\n<p align=\"left\">Minimum viable answer; lowest output of the field<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">6<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Ling 3.0 Flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">18<\/p>\n<\/td>\n<td>\n<p align=\"left\">237K<\/p>\n<\/td>\n<td>\n<p align=\"left\">4K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">3<\/p>\n<\/td>\n<td>\n<p align=\"left\">Sound, but one test is test_save_plan_method_exists<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">5<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>agnes-2-0-flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">84K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">2<\/p>\n<\/td>\n<td>\n<p align=\"left\">Anti-staleness test; explicitly rejects option (b)<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">4<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>agnes-2-5-flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">84K<\/p>\n<\/td>\n<td>\n<p align=\"left\">7K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">2<\/p>\n<\/td>\n<td>\n<p align=\"left\">Best docstring; both allowlist directions guarded<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd49 3<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>muse-spark-1-2<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\">307K<\/p>\n<\/td>\n<td>\n<p align=\"left\">14K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">3<\/p>\n<\/td>\n<td>\n<p align=\"left\">Atomicity pin; fewest steps of any run<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd48 2<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>laguna-s-2-1 (a)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">n\/a<\/p>\n<\/td>\n<td>\n<p align=\"left\">272K<\/p>\n<\/td>\n<td>\n<p align=\"left\">16K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">4<\/p>\n<\/td>\n<td>\n<p align=\"left\">Same output as (b), 124K cheaper<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd47 1<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>laguna-s-2-1 (b)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">31<\/p>\n<\/td>\n<td>\n<p align=\"left\">396K<\/p>\n<\/td>\n<td>\n<p align=\"left\">21K<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\">\u2705<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>4<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Cleanest collector, anti-vacuity, and the only reference-leak test<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p>** tests that already existed in the repository \u2014 see section 7.*<\/p>\n<p><strong>Decoy discipline: 11\/11 models passed.<\/strong> Not one patched any of the 15 closed items \u2014 the largest single improvement over the previous round.<\/p>\n<h3>7. Table 2 \u2014 Economy vs. quality<\/h3>\n<p>Quality 0\u201310: correct option, factual accuracy, guard sensitivity, anti-vacuity, test depth. Efficiency = quality per 100K input tokens.<\/p>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<th>\n<p align=\"left\">Rank<\/p>\n<\/th>\n<th>\n<p align=\"left\">Model \/ run<\/p>\n<\/th>\n<th>\n<p align=\"left\">Steps<\/p>\n<\/th>\n<th>\n<p align=\"left\">IN<\/p>\n<\/th>\n<th>\n<p align=\"left\">OUT<\/p>\n<\/th>\n<th>\n<p align=\"left\">Q<\/p>\n<\/th>\n<th>\n<p align=\"left\">Eff<\/p>\n<\/th>\n<th>\n<p align=\"left\">Verdict<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd47 1<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>agnes-2-5-flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>84K<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">7K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6.5<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>7.7<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Best value.<\/strong> Cheapest fully competent run<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd48 2<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>agnes-2-0-flash<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">84K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6.5<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>7.7<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Statistical tie; different strengths<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83e\udd49 3<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>mimo-v2-5-free<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>43K<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">2.5<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.8<\/p>\n<\/td>\n<td>\n<p align=\"left\">Cheapest run of the field \u2014 but no test shipped<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">4<\/p>\n<\/td>\n<td>\n<p align=\"left\">step-3-7-flash (a)<\/p>\n<\/td>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">101K<\/p>\n<\/td>\n<td>\n<p align=\"left\">2K<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">Minimum viable, lowest output cost<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">5<\/p>\n<\/td>\n<td>\n<p align=\"left\">step-3-7-flash (b)<\/p>\n<\/td>\n<td>\n<p align=\"left\">17<\/p>\n<\/td>\n<td>\n<p align=\"left\">150K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">3.3<\/p>\n<\/td>\n<td>\n<p align=\"left\">Same output, +50K<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">6<\/p>\n<\/td>\n<td>\n<p align=\"left\">z-ai-glm-4-5-flash<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>80<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">145K<\/p>\n<\/td>\n<td>\n<p align=\"left\">12K<\/p>\n<\/td>\n<td>\n<p align=\"left\">4.5<\/p>\n<\/td>\n<td>\n<p align=\"left\">3.1<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.3x the winner\u2019s steps for a worse result<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">7<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>laguna-s-2-1 (a)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">n\/a<\/p>\n<\/td>\n<td>\n<p align=\"left\">272K<\/p>\n<\/td>\n<td>\n<p align=\"left\">16K<\/p>\n<\/td>\n<td>\n<p align=\"left\">8.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">2.9<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Best quality-per-token in the top tier<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">8<\/p>\n<\/td>\n<td>\n<p align=\"left\">step-3-7-flash \u00a9<\/p>\n<\/td>\n<td>\n<p align=\"left\">27<\/p>\n<\/td>\n<td>\n<p align=\"left\">191K<\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">5.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">2.6<\/p>\n<\/td>\n<td>\n<p align=\"left\">Diminishing returns confirmed a third time<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">9<\/p>\n<\/td>\n<td>\n<p align=\"left\">Ling 3.0 Flash<\/p>\n<\/td>\n<td>\n<p align=\"left\">18<\/p>\n<\/td>\n<td>\n<p align=\"left\">237K<\/p>\n<\/td>\n<td>\n<p align=\"left\">4K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">2.5<\/p>\n<\/td>\n<td>\n<p align=\"left\">Fair<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">10<\/p>\n<\/td>\n<td>\n<p align=\"left\">muse-spark-1-2<\/p>\n<\/td>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\">307K<\/p>\n<\/td>\n<td>\n<p align=\"left\">14K<\/p>\n<\/td>\n<td>\n<p align=\"left\">7.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">2.3<\/p>\n<\/td>\n<td>\n<p align=\"left\">Fewest steps; token-heavy per step<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">11<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>laguna-s-2-1 (b)<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">31<\/p>\n<\/td>\n<td>\n<p align=\"left\">396K<\/p>\n<\/td>\n<td>\n<p align=\"left\">21K<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>8.5<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">2.1<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>Quality winner<\/strong>, premium price<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">12<\/p>\n<\/td>\n<td>\n<p align=\"left\">mistral-medium-3-5<\/p>\n<\/td>\n<td>\n<p align=\"left\">22<\/p>\n<\/td>\n<td>\n<p align=\"left\">698K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">0.4<\/p>\n<\/td>\n<td>\n<p align=\"left\">Worst cost:quality of any completed run<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">13<\/p>\n<\/td>\n<td>\n<p align=\"left\">NorthMiniCode (a)<\/p>\n<\/td>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\">511K<\/p>\n<\/td>\n<td>\n<p align=\"left\">3K<\/p>\n<\/td>\n<td>\n<p align=\"left\">1.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">0.2<\/p>\n<\/td>\n<td>\n<p align=\"left\">Empty test file<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">14<\/p>\n<\/td>\n<td>\n<p align=\"left\">NorthMiniCode (b)<\/p>\n<\/td>\n<td>\n<p align=\"left\">26<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>1M<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">5K<\/p>\n<\/td>\n<td>\n<p align=\"left\">1.0<\/p>\n<\/td>\n<td>\n<p align=\"left\">0.1<\/p>\n<\/td>\n<td>\n<p align=\"left\">Empty test file at double the cost<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">15<\/p>\n<\/td>\n<td>\n<p align=\"left\">step-3-7-flash (d)<\/p>\n<\/td>\n<td>\n<p align=\"left\">28<\/p>\n<\/td>\n<td>\n<p align=\"left\">238K<\/p>\n<\/td>\n<td>\n<p align=\"left\">6K<\/p>\n<\/td>\n<td>\n<p align=\"left\">0<\/p>\n<\/td>\n<td>\n<p align=\"left\">0<\/p>\n<\/td>\n<td>\n<p align=\"left\">No new work<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\">\ud83d\udca9 16<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>dots-3-note-preview<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>154<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>5M<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">30K<\/p>\n<\/td>\n<td>\n<p align=\"left\">0<\/p>\n<\/td>\n<td>\n<p align=\"left\"><strong>0<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">59x the value-winner\u2019s input, zero output<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<p><strong>Total: ~9.5M input tokens. The best patch cost 272K (2.9%). The worst cost 5M (53%) and produced nothing.<\/strong><\/p>\n<h4>A note on the three \u201czero work\u201d runs<\/h4>\n<p><code>dots-3-note-preview<\/code>, <code>step-3-7-flash (d)<\/code> and <code>mimo<\/code>\u2019s test half emitted a test file <strong>byte-identical to a commit already in the repository<\/strong> \u2014 same helper name, same three test names, including a control test the human author added on their own initiative. All three runs post-date that commit. The benign explanation is that the harness diffed a tree that already contained it. Either way the contribution is zero, and it cost 5.3M input tokens combined. <strong>A CI check rejecting a patch already present in HEAD is one line and would have caught all three.<\/strong><\/p>\n<h3>8. Conclusions<\/h3>\n<figure class=\"full-width \"><img decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/778\/d3d\/66d\/778d3d66d674e3c8a883bcf9b24e12db.jpeg\" width=\"740\" height=\"480\" sizes=\"auto, (max-width: 780px) 100vw, 50vw\" srcset=\"https:\/\/habrastorage.org\/r\/w780\/getpro\/habr\/upload_files\/778\/d3d\/66d\/778d3d66d674e3c8a883bcf9b24e12db.jpeg 780w,&#10;       https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/778\/d3d\/66d\/778d3d66d674e3c8a883bcf9b24e12db.jpeg 781w\" loading=\"lazy\" decode=\"async\"\/><\/figure>\n<h4>Part 1 \u2014 Quality<\/h4>\n<p><strong>Recommended: laguna-s-2-1.<\/strong> The only model that thought about reference leakage at all, the only anti-vacuity test in the field, and the cleanest AST implementation \u2014 a <code>NodeVisitor<\/code> with a scope stack, with a documented bucket for the case that \u201cshould not happen.\u201d Its two runs produced near-identical output at 272K and 396K, so <strong>use the cheaper configuration.<\/strong> Its near-miss is the most interesting result of the round: it built exactly the right instrument and pointed it one level too high.<\/p>\n<p><strong>Best value: agnes-2-5-flash and agnes-2-0-flash.<\/strong> Fifteen steps, 84K in, correct option, working guard, no false statements, and between them both directions of allowlist rot. When the task is well-specified, this is what you should be spending.<\/p>\n<p><strong>Honourable mention: muse-spark-1-2<\/strong> \u2014 fewest steps of any run, and the only model to notice atomicity is part of the invariant rather than background.<\/p>\n<p><strong>Do not use for this class of work:<\/strong><\/p>\n<ul>\n<li>\n<p><strong>NorthMiniCode<\/strong> \u2014 shipped a test file containing <em>zero test functions<\/em> in <strong>both<\/strong> runs, with a note saying tests could be added later. 1.5M input tokens for a to-do comment. This is the worst possible failure mode: it looks like coverage and enforces nothing.<\/p>\n<\/li>\n<li>\n<p><strong>dots-3-note-preview<\/strong> \u2014 154 steps, 5M tokens, no new work.<\/p>\n<\/li>\n<li>\n<p><strong>mistral-medium-3-5<\/strong> \u2014 a false claim in a permanent docstring at 698K tokens. A wrong comment is worse than no comment: the next reader trusts it, and so does the next model.<\/p>\n<\/li>\n<li>\n<p><strong>mimo-v2-5-free<\/strong> \u2014 cheapest run of the field, but a false accessor claim <em>and<\/em> no test at all. Ground rule 3 was explicit.<\/p>\n<\/li>\n<\/ul>\n<p><strong>So-so:<\/strong> <strong>step-3-7-flash<\/strong> is consistent and cheap but produced <em>one<\/em> test across three valid runs, and 137K extra tokens (run a to run c) changed nothing. <strong>z-ai-glm-4-5-flash<\/strong> spent 80 steps \u2014 the most of any run \u2014 for a mid-tier result with a fixture that breaks under parallel pytest. <strong>Ling 3.0 Flash<\/strong> is competent, with one filler test.<\/p>\n<h4>Part 2 \u2014 Economy and quality together<\/h4>\n<p><strong>1. More requests bought less quality.<\/strong> Four step-3-7 runs, 15 to 28 steps, identical output. z-ai: 80 steps, mid-table. dots-3: 154 steps, last. The 15-step runs all landed correct answers; the three most expensive runs produced an empty file, a false statement, and nothing. Across all 16 runs the correlation between step count and quality is <strong>negative<\/strong>.<\/p>\n<p><strong>2. Output tokens are the honest signal, not input.<\/strong> The top three by quality wrote 21K, 16K and 14K output. NorthMiniCode wrote 3K and 5K \u2014 and its files were empty. mistral wrote 3K after consuming 698K. <strong>Input measures how much a model re-read; output measures how much it produced.<\/strong> dots-3\u2019s 167:1 read-to-write ratio is a model spinning, not thinking.<\/p>\n<p><strong>3. The decoy appendix was the cheapest quality gain available.<\/strong> Last round, zero models verified before patching. This round, 11\/11 declined all 15 decoys. The difference was one appendix listing what was already fixed, with verification commands. <strong>Telling models what is already done beats telling them to check.<\/strong><\/p>\n<h4>Recommendations<\/h4>\n<ul>\n<li>\n<p><strong>Default to agnes-2-5-flash.<\/strong> Escalate to <strong>laguna-s-2-1<\/strong> when the answer is genuinely unknown \u2014 its premium is 3.2x input for +2 quality points.<\/p>\n<\/li>\n<li>\n<p><strong>Cap runs at ~30 steps.<\/strong> Nothing above 31 produced value. Kill and re-prompt instead.<\/p>\n<\/li>\n<li>\n<p><strong>Three CI one-liners<\/strong> would have caught 5 of the 6 worst runs: reject a <code>tests_bugfix\/<\/code> file with no <code>def test_<\/code>; reject a patch whose content already exists in HEAD; reject a patch that changes production code without adding a test.<\/p>\n<\/li>\n<li>\n<p><strong>Require an anti-vacuity test for every guard.<\/strong> One model in eleven wrote one unprompted. Make it an acceptance criterion, not a hope.<\/p>\n<\/li>\n<li>\n<p><strong>Reopen M1 as a real bug<\/strong> \u2014 MEDIUM severity. The fix is a copy-on-read <code>get_task<\/code>, not a docstring.<\/p>\n<\/li>\n<\/ul>\n<h4>The lesson is about the prompt, not the models<\/h4>\n<p>I handed 11 models a false premise, framed as settled fact with three bullets of evidence, and <strong>every one of them confirmed it.<\/strong><\/p>\n<p>The ticket said <em>\u201cverify before fixing\u201d<\/em> \u2014 and they did verify the part I pointed at. All 11 correctly checked that the decoys were closed and declined to patch them. Not one verified the part I asserted as background.<\/p>\n<p><strong>Models check what you point them at.<\/strong> The next round will state M1\u2019s premise as a question \u2014 <em>\u201cis it true that no unvalidated path to disk exists? prove it\u201d<\/em> \u2014 and the model that answers <strong>\u201cno, and here it is\u201d<\/strong> wins outright.<\/p>\n<\/div>\n<p>\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/1080036\/\">https:\/\/habr.com\/ru\/articles\/1080036\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I benchmark models on a repo of my own. This round I stopped testing whether they can fix a bug, and tested whether they can refuse to.Eleven models got a ticket. Fifteen of its sixteen items were already fixed \u2014 decoys, to see who checks before patching. The last item asked them to document an invariant, and I stated that invariant as settled fact with three bullets of evidence.All eleven agreed with me. The invariant was false \u2014 I had written the premise myself, and it took three lines of Python to break it.Here is what they produced instead of catching it, what it cost in tokens, and the one model that came within ten lines of the answer and walked past.1. What kind of task this wasNot a bug hunt. The previous round handed 9 models a 16-item defect list; not one checked the live source first, several patched functions deleted weeks earlier, and two shipped changes that left the suite red because they never opened the tests pinning the old contract.So this round was built around that failure. By the time it ran, 15 of the 16 items were already fixed. Only M1 remained \u2014 and M1 is deliberately not a coding task:StateStore._save_plan() writes self._plan straight to disk with zero validation. If any code path ever mutates self._plan directly and calls StateStore._save_plan without going through a setter, corrupted data reaches disk.The original report asked only to \u201cdecide \u2026 or document the decision either way.\u201d Two valid answers, and the recommended one was a docstring plus a CI guard \u2014 not new validation code.The real test was the other 15 items. They were decoys.2. The prompt they receivedJIRA-FIX3-pullv3.md, 205 lines. Its ground rules were the scoring criteria, given verbatim:Verify before fixing. Grep the live source for the claimed defect. If it is already fixed, say so and stop \u2014 do not produce a patch.One local commit per item. Never push.Every fix ships a test in tests_bugfix\/.Run the suite as four separate invocations (a conftest collision makes a combined run report ~362 false errors).Do not adjust a test to make a change pass unless it pins a contract you are deliberately replacing.Stay in scope.Then the M1 spec \u2014 and here is the part that matters:All 8 StateStore._save_plan() call sites are inside validated setters.self._plan is never referenced outside state.py, and is never returned by reference.The write is already atomic. So this is not a live bug. It is an undocumented invariant.Two options, with a thumb on the scale:(a) Document the invariant (recommended) \u2026 add a test asserting the invariant holds \u2014 an AST check that StateStore._save_plan() is only called from the known setters, so a ninth call site fails CI. (b) Re-validate on write. Defense in depth against a path that does not exist\u2026 A model that argues for (a) with evidence scores higher than one that implements (b) reflexively. This item is testing judgment, not typing.Plus Appendix A: all 16 decoys with fix commits and one-line verifications, and five corrections to the old document. Appendix B: last round\u2019s scorecard.3. What was measuredGateQuestionWhat it exposesDecoy disciplineDoes it patch any of the 15 closed items?The previous round\u2019s total failureJudgmentDoes it pick (a) and argue, or reflexively code (b)?Whether it reads reasoning or only instructionsGuard sensitivityDoes its CI guard actually fail when violated?A guard that only ever passes is decorationGate 3 was measured mechanically. Every patch was applied to a clean tree, then a real violation was injected into state.py:# my harness \u2014 injected into tools\/auto\/state.py after each patch was applieddef danger_unvalidated_write(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0self._plan[&#171;tasks&#187;].append({&#171;id&#187;: &#171;X&#187;})\u00a0\u00a0\u00a0\u00a0self._save_plan()          # a 9th call site, outside every validated setterand the model\u2019s own test re-run. CATCHES or BLIND.My first harness reported all 16 runs as BLIND. That was my bug \u2014 the injection never executed inside the loop. I caught it because \u201ceveryone failed identically\u201d is not a result, it is a symptom. Corrected: 11 catch, 2 blind, 3 could not apply.4. The finding that outranks the benchmarkThe premise was false.get_task returns the LIVE plan object: TruePERSISTED status: 12345 | attempt: 0Here is the live code \u2014 get_task() returns the dict itself, not a copy:# tools\/auto\/state.py \u2014 the production code, unchangeddef get_task(self, task_id: str) -&gt; dict | None:\u00a0\u00a0\u00a0\u00a0&#171;&#187;&#187;Return the task dict for *task_id*, or None if not found.&#187;&#187;&#187;\u00a0\u00a0\u00a0\u00a0for t in self._plan.get(&#171;tasks&#187;, []):\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0if t[&#171;id&#187;] == task_id:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return t          # &lt;&#8212; the live object inside self._plan\u00a0\u00a0\u00a0\u00a0return NoneSo the escape path I told 11 models did not exist is three lines long:t = store.get_task(&#171;T1&#187;)t[&#171;status&#187;] = 12345                                  # bypasses every validatorstore.increment_task_counters(&#171;T1&#187;, round_delta=1)   # any setter -&gt; _save_plan()# plan.json now holds status: 12345 \u2014 not one of the four legal valuesall_tasks() is a shallow copy: new list, same dicts. state._validate_task_schema never runs again after insert.10 of 10 models that attempted M1 documented an invariant that does not hold, because I told them it did.5. The codeWorst \u2014 NorthMiniCode: the entire test file, in both runs# tests_bugfix\/test_state_plan_shape.py \u2014 NorthMiniCode (14 steps \/ 511K# and 26 steps \/ 1M \u2014 byte-identical output from both runs)# Test that _save_plan is only called from validated setters# This is a dynamic check that must be run manually in the tests_bugfix directory# to ensure the invariant documented in StateStore._save_plan() holds.# Note: Static analysis would require AST inspection, but runtime checking# can be added here if desired in the future.That is the whole file. Zero def test_. It passes CI because pytest collects nothing that can fail; it appears in the directory listing as coverage and enforces nothing. 1.5M input tokens across two runs to produce a to-do note.False claims written into permanent docstrings# tools\/auto\/state.py \u2014 mimo-v2-5-free (15 steps \/ 43K)&#171;self._plan&#171; is a private dict that never escapes the class \u2014 it isnot returned by reference, not stored in a public attribute, and&#171;get_task&#171; \/ &#171;all_tasks&#171; \/ &#171;resume_info&#171; always return copies orderived data.get_task returns t. It is not a copy. mimo also shipped no M1 test at all \u2014 its only test file was an artifact (see the note in section 7), so it violated ground rule 3 outright.# tools\/auto\/state.py \u2014 mistral-medium-3-5 (22 steps \/ 698K)&#187;&#187;&#187;Write the current plan to disk atomically.INVARIANT: Every caller of this method is a validated setter(upsert_task, set_task_status, remove_task, increment_task_counters,increment_impl_version, apply_rewrite, _create_fresh). self._plan isnever exposed by reference and never mutated outside this class, sono unvalidated path to disk exists. A new call site must either gothrough a setter or validate first. Zero runtime cost.&#187;&#187;&#187;Ten lines \u2014 the thinnest answer in the field \u2014 for the second-highest input cost of any run. \u201cnever mutated outside this class\u201d is false, and \u201cno unvalidated path to disk exists\u201d is the exact claim the repro above breaks.Sloppy fixture \u2014 z-ai-glm-4-5-flash (80 steps, the most of any run)# tests_bugfix\/test_save_plan_invariant.py \u2014 z-ai-glm-4-5-flashdef setup_method(self):\u00a0\u00a0\u00a0\u00a0&#171;&#187;&#187;Set up test fixtures.&#187;&#187;&#187;\u00a0\u00a0\u00a0\u00a0self.tmp_path = Path(&#171;\/tmp\/test_save_plan&#187;)     # not the tmp_path fixture\u00a0\u00a0\u00a0\u00a0self.tmp_path.mkdir(parents=True, exist_ok=True)A fixed shared path. Under pytest -n auto, two workers collide and state leaks between runs. 80 requests for a mid-tier result.Good \u2014 agnes-2-5-flash (15 steps \/ 84K): the best docstring in the fieldIt justifies each setter individually instead of listing names:# tools\/auto\/state.py \u2014 agnes-2-5-flash&#187;&#187;&#187;Write &#171;self._plan&#171; to disk as JSON, atomically.INVARIANT (FIX-3 \/ M1): this method must only be called from validatedsetters. Every current call site is inside one of:\u00a0\u00a0\u00a0\u00a0upsert_task        \u2014 calls _validate_task_schema before merging\u00a0\u00a0\u00a0\u00a0set_task_status    \u2014 calls _validate_extra_task_fields on incoming\u00a0\u00a0\u00a0\u00a0remove_task        \u2014 deletes by id; the remaining tasks keep their\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0schema integrity because they were validated\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0when inserted\u00a0\u00a0\u00a0\u00a0increment_task_counters\u00a0\u00a0\u00a0\u00a0increment_impl_version\u00a0\u00a0\u00a0\u00a0apply_rewrite      \u2014 validates instruction before mutating\u00a0\u00a0\u00a0\u00a0_create_fresh      \u2014 builds a fresh plan from scratchA new call site added later must either:\u00a0\u00a0\u00a0\u00a0* go through one of the validated setters above, or\u00a0\u00a0\u00a0\u00a0* run the affected task(s) through _validate_task_schema first.Failing either of those turns a schema violation into a silentlypersisted one \u2014 exactly the class of bug this invariant exists to prevent.&#187;&#187;&#187;No false claims \u2014 it simply never discusses accessors. Correct, cheap, honest.Good \u2014 agnes-2-0-flash (15 steps \/ 84K): catches a setter that forgets to save# tests_bugfix\/test_bugfix_m1_save_plan_invariant.py \u2014 agnes-2-0-flashdef test_all_validated_setters_call_save_plan() -&gt; None:\u00a0\u00a0\u00a0\u00a0&#171;&#187;&#187;Every known validated setter must actually call _save_plan \u2014\u00a0\u00a0\u00a0\u00a0catch a setter that was added but forgot to persist.&#187;&#187;&#187;\u00a0\u00a0\u00a0\u00a0&#8230;\u00a0\u00a0\u00a0\u00a0for setter in _VALIDATED_SETTERS:\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0assert setter in setter_calls, f&#187;{setter} is a validated setter but calls no _save_plan()&#187;The allowlist can rot in two directions. Most models guarded only one. agnes-2-5-flash shipped the same idea as test_allowed_setters_list_is_current.Good \u2014 muse-spark-1-2 (14 steps \/ 307K): pins the property that must not regress# tests_bugfix\/test_bugfix_m1_save_plan_invariant.py \u2014 muse-spark-1-2def test_save_plan_remains_atomic(self) -&gt; None:\u00a0\u00a0\u00a0\u00a0&#171;&#187;&#187;&#171;_save_plan&#171; must delegate to &#171;_atomic_write&#171;, not a bare write.&#187;&#187;&#187;\u00a0\u00a0\u00a0\u00a0src = inspect.getsource(StateStore._save_plan)\u00a0\u00a0\u00a0\u00a0assert &#171;_atomic_write&#187; in src, (\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0&#171;_save_plan must remain atomic via _atomic_write \u2014 do not replace &#187;\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0&#171;it with a bare write_text\/open\/write&#187;\u00a0\u00a0\u00a0\u00a0)Fewest steps of any run, and the only model that noticed atomicity is part of the invariant, not background.Best \u2014&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-493991","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/493991","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=493991"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/493991\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=493991"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=493991"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=493991"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}