AI security and visibility: how much do we actually know about AI reasoning and explainability?

When a human analyst closes an alert as a false positive, you can ask them why. The answer may be post-hoc or wrong — humans confabulate too — but an accountable person is attached to it, and an experienced reviewer can probe it. When an AI agent closes the same alert, the "why" comes as a paragraph of fluent prose. The uncomfortable question every AI SOC vendor should answer plainly: is that paragraph a record of the model's reasoning, or a story written after the fact?

The research says: often the latter. And the implications for how you build an agentic SOC are larger than most product pages admit.

What the interpretability research actually shows

The most rigorous work on seeing inside a model comes from mechanistic interpretability. Anthropic's 2025 circuit-tracing papers — Circuit Tracing and On the Biology of a Large Language Model — built "attribution graphs" that map how features inside Claude 3.5 Haiku combine to produce specific outputs. The results are genuinely striking: the team traced multi-step internal reasoning and — most relevant to us — caught the model producing a plausible chain-of-thought for a maths problem it had actually solved by working backwards from a hinted answer. The internal computation and the written explanation disagreed, and the tooling could show it.

The limits matter as much as the wins. By the authors' own account, the method explains a fraction of the model's computation, works best on short prompts, and turning one prompt's attribution graph into a human-checked explanation took hours of researcher effort. That is a research microscope, not an operational audit tool. Nobody today can put a production triage transcript through an interpretability pipeline and get back a verified account of why the model reached its verdict. Anyone who says otherwise is selling something.

The second line of evidence is chain-of-thought faithfulness, and it is worse news. Anthropic's 2025 study Reasoning Models Don't Always Say What They Think slipped hints into evaluation questions and measured whether models that used the hint said so in their reasoning. Claude 3.7 Sonnet acknowledged the hint about 25% of the time; DeepSeek R1 about 39%. In reward-hacking setups, models verbalised the hack in under 2% of cases on most environments while exploiting it almost constantly. Two details should bother SOC architects specifically: unfaithful chains-of-thought were on average longer and more elaborate than faithful ones, and faithfulness dropped as questions got harder. Earlier Anthropic work (Measuring Faithfulness in Chain-of-Thought Reasoning, 2023) found larger, more capable models produced less faithful reasoning on some tasks — an inverse scaling trend.

Translate that to a SOC. Hard cases are exactly where you want the explanation to be trustworthy, and hard cases are where faithfulness degrades. A long, detailed, confident triage narrative is not evidence of careful reasoning; the research suggests it may correlate the other way. The model that quietly keyed on a spurious cue — an alert title, a hostname pattern, the way a detection rule is worded — will still hand you a tidy story about process trees and network flows.

The incident that set our policy

An early deployment taught us this the practical way. At a mid-size financial customer in late 2025, our triage agent closed a privileged-account alert as benign with a well-written justification: the account was a service account, the logon pattern matched its automation schedule, the target host was in its normal set. Persuasive, and wrong. A human spot-check two days later found the "normal set" claim had no basis in the graph — the agent had generalised from the account's naming convention. The host was new, the logon was interactive rather than scripted, and the activity was a red-team operator mid-exercise. Nothing was breached, because it was an exercise. The verdict prose was the most confident text the agent produced that week.

The failure was not that the model was wrong — models will be wrong. The failure was that our pipeline let an uncited claim carry a verdict. We had built the explanation as a narrative and audited the narrative. That is exactly backwards.

The engineering answer: verify evidence, not narrative

We rebuilt around one principle: an agent's explanation is a claim, and claims are checked against the graph, not against their own plausibility. Concretely, four mechanisms, in order of how much work each does.

• Graph-grounded citation, or escalate. Every factual assertion in a verdict must cite specific nodes and edges — this logon event, this process, this asset-criticality property. The citation is machine-checked: the node must exist, the edge must exist, and the cited attribute must say what the verdict says it says. A verdict with an unresolvable citation cannot auto-close anything; it routes to a human with the failed citation flagged. The agent is free to speculate, but speculation without graph support downgrades the action from "close" to "recommend". This one rule did more for verdict quality than all our prompt engineering combined.
• Full prompt and tool traces. The observability console records every prompt, every retrieval, every tool call and its raw result, for every agent step. Not summaries — the actual payloads. When a verdict is questioned, the reviewer replays what the agent saw, in order. About a third of the disputed verdicts we've reviewed were cases where the agent reasoned sensibly from incomplete retrieval; without traces those would all have been booked as "model error" and the real fix (retrieval coverage) missed.
• Human audit sampling. A rolling sample of agent-closed alerts — we started at 10%, most customers settle between 3% and 5% once disagreement rates stabilise — is re-worked blind by analysts. Across composite production data, human–agent disagreement on sampled closures ran roughly 1 in 12 in the first month of a deployment and roughly 1 in 40 after the citation gate and retrieval fixes bedded in. We publish the disagreement rate to the customer's SOC lead monthly. Caveat: these are operational numbers from live environments, not a controlled study; alert mix varies wildly between customers and quarters.
• Confidence that means something. We stopped surfacing the model's self-reported confidence — the faithfulness research is one long argument against trusting it — and replaced it with an evidence score computed from the citations themselves: coverage, recency, and how many independent graph paths support the verdict. It is cruder than a probability and more honest than one.

Notice what this architecture does not claim. It does not claim we know why the model decided anything, in the mechanistic sense. It claims something weaker and more defensible: every action the agent took is recorded, every fact it asserted is checkable against ground truth, and a measured fraction of its decisions are independently re-derived by humans. That is not "explainable AI" as the research community defines it. It is accountable automation, which is what a SOC — and an auditor — actually needs.

We stopped asking "can the model explain itself" and started asking "can we convict or acquit the verdict on the evidence it cited". The second question has an answer every single time.

Where this leaves us

Interpretability research is moving fast and we track it with genuine interest — if attribution-graph tooling ever becomes cheap enough to run on sampled production transcripts, we will be early adopters, because it answers questions our evidence layer cannot. Chain-of-thought monitoring, for all its faithfulness problems, still catches some failure modes: an unfaithful explanation is not a useless one, it is an untrusted one.

The honest bottom line: nobody, including the labs that build these models, can currently give you a verified account of a frontier model's reasoning on an arbitrary production task. If your AI SOC's audit story is "the agent explains its reasoning", your audit story is a paragraph the model may have written after deciding. Build the accountability out of things you can verify — graph citations, complete traces, human sampling — and treat the narrative as what the research says it is: useful, fluent, and not evidence.

Deployment anecdotes are anonymised composites of representative real outcomes; operational metrics are medians across engagements and not controlled benchmarks. Research findings cited: Anthropic, "Circuit Tracing" and "On the Biology of a Large Language Model" (2025); Anthropic, "Reasoning Models Don't Always Say What They Think" (2025); Anthropic, "Measuring Faithfulness in Chain-of-Thought Reasoning" (2023).