{"id":499863,"date":"2026-10-22T22:01:18","date_gmt":"2026-10-22T19:01:18","guid":{"rendered":"https:\/\/savepearlharbor.com\/?p=499863"},"modified":"2026-10-22T22:01:18","modified_gmt":"2026-10-22T19:01:18","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=499863","title":{"rendered":"Is LLM Interpretability Proven?"},"content":{"rendered":"
\n

A recently published research essay claims that the difficulty of neural network interpretability has been overstated. In his essay<\/a> on goodfire.com<\/a>, Nick Cammarata argues that neuron polysemanticity is caused not by superposition but by ordinary, mostly fixable issues \u2014 and that most neurons, even in massive models, perform a single, understandable job. The author himself calls the text a working hypothesis: the technical paper with a full description of the experiments has not been released yet. However, as usual, a bold headline and strong claims have sparked numerous retellings in which the hypothesis has turned into established fact.<\/p>\n

Below is a quick look at this hypothesis.<\/p>\n

Spot-checking Cammarata\u2019s claims<\/h3>\n

Summarizing the essay is probably unnecessary. It is written clearly and persuasively, so it is easy to follow the arguments and the preliminary description of the experiments on your own.<\/p>\n

First, it is worth noting that Goodfire is a commercial company that offers, among other things, LLM verification as a service, with Silico as its core tool. This does not mean the essay is wrong, but there are far too many examples of business pressure pushing teams to draw conclusions too rigidly or too quickly.<\/p>\n

Second, the neuron audit was conducted on Qwen3-32B, with hypotheses generated and checked with the help of AI agents. In other words, LLMs were used to help prove that LLMs are interpretable. It looks amusingly circular, though drawing conclusions here is also premature until the full experimental details are published.<\/p>\n

Cammarata\u2019s claim that weak activations are secondary \u201crecursive generalization\u201d rests on an ablation measured by average accuracy: removing the weakest 75% of activations (weak75) reduces accuracy only from 72% to 71%. The author notes that roughly one in ten answers changes in the process, but considers the lost distinctions minor \u2014 fine distinctions between classes the network already confuses.<\/p>\n

This claim is testable, so I ran an analogous experiment on a small network. My favorite task: classifying whether a sequence is sorted. Let\u2019s see just how secondary the activation tail really is.<\/p>\n

On my MLP-512 model, under an analogous weak75 protocol, accuracy on the standard test set actually increased<\/em>, from 97.5% to 98.3% \u2014 which at first glance fully supports Cammarata. However, on boundary cases (sequences where only two adjacent elements are swapped), ablating the tail made accuracy plummet from 93.1% to 72.4%. Since random guessing gives 50% here, the network lost nearly half (48%) of its above-chance competence. Recall on detecting violations was roughly halved, from ~88% to ~44%.<\/p>\n

The takeaway: weak activations are precisely where class boundaries live \u2014 the boundaries that determine whether a model can tell closely related cases apart. An important caveat: randomly ablating the same mass of activations has almost the same effect. So the boundaries are held by the weak, distributed mass of activations as a whole, of which the tail is the bulk.<\/p>\n

In essence, Cammarata tested typical cases rather than boundaries.<\/p>\n

Conclusion<\/h3>\n

His own evaluation criterion (ablating the weakest 75% of activations and measuring average accuracy) can create an illusion that the network remains intact. Cammarata sees that the tail handles subtle cases but dismisses this as a minor detail \u2014 yet on boundary cases, removing that \u201cminor detail\u201d collapses the network\u2019s boundary competence.<\/p>\n

In other words, Cammarata\u2019s networks may look clean simply because he checks them for cleanliness with a metric in which boundaries carry almost no weight.<\/p>\n

In any case, a full assessment of Cammarata\u2019s claims will only be possible once the complete technical report is published. Though, as usual, that hasn\u2019t stopped the hype.<\/p>\n

Code and data: <\/em>git<\/em><\/a><\/p>\n<\/div>\n

\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1098110\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

A recently published research essay claims that the difficulty of neural network interpretability has been overstated. In his essay on goodfire.com, Nick Cammarata argues that neuron polysemanticity is caused not by superposition but by ordinary, mostly fixable issues \u2014 and that most neurons, even in massive models, perform a single, understandable job. The author himself calls the text a working hypothesis: the technical paper with a full description of the experiments has not been released yet. However, as usual, a bold headline and strong claims have sparked numerous retellings in which the hypothesis has turned into established fact.Below is a quick look at this hypothesis.Spot-checking Cammarata\u2019s claimsSummarizing the essay is probably unnecessary. It is written clearly and persuasively, so it is easy to follow the arguments and the preliminary description of the experiments on your own.First, it is worth noting that Goodfire is a commercial company that offers, among other things, LLM verification as a service, with Silico as its core tool. This does not mean the essay is wrong, but there are far too many examples of business pressure pushing teams to draw conclusions too rigidly or too quickly.Second, the neuron audit was conducted on Qwen3-32B, with hypotheses generated and checked with the help of AI agents. In other words, LLMs were used to help prove that LLMs are interpretable. It looks amusingly circular, though drawing conclusions here is also premature until the full experimental details are published.Cammarata\u2019s claim that weak activations are secondary \u201crecursive generalization\u201d rests on an ablation measured by average accuracy: removing the weakest 75% of activations (weak75) reduces accuracy only from 72% to 71%. The author notes that roughly one in ten answers changes in the process, but considers the lost distinctions minor \u2014 fine distinctions between classes the network already confuses.This claim is testable, so I ran an analogous experiment on a small network. My favorite task: classifying whether a sequence is sorted. Let\u2019s see just how secondary the activation tail really is.On my MLP-512 model, under an analogous weak75 protocol, accuracy on the standard test set actually increased, from 97.5% to 98.3% \u2014 which at first glance fully supports Cammarata. However, on boundary cases (sequences where only two adjacent elements are swapped), ablating the tail made accuracy plummet from 93.1% to 72.4%. Since random guessing gives 50% here, the network lost nearly half (48%) of its above-chance competence. Recall on detecting violations was roughly halved, from ~88% to ~44%.The takeaway: weak activations are precisely where class boundaries live \u2014 the boundaries that determine whether a model can tell closely related cases apart. An important caveat: randomly ablating the same mass of activations has almost the same effect. So the boundaries are held by the weak, distributed mass of activations as a whole, of which the tail is the bulk.In essence, Cammarata tested typical cases rather than boundaries.ConclusionHis own evaluation criterion (ablating the weakest 75% of activations and measuring average accuracy) can create an illusion that the network remains intact. Cammarata sees that the tail handles subtle cases but dismisses this as a minor detail \u2014 yet on boundary cases, removing that \u201cminor detail\u201d collapses the network\u2019s boundary competence.In other words, Cammarata\u2019s networks may look clean simply because he checks them for cleanliness with a metric in which boundaries carry almost no weight.In any case, a full assessment of Cammarata\u2019s claims will only be possible once the complete technical report is published. Though, as usual, that hasn\u2019t stopped the hype.Code and data: git\u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 https:\/\/habr.com\/ru\/articles\/1098110\/<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-499863","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/499863","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=499863"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/499863\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=499863"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=499863"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=499863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}