Every quarter a customer asks some version of: "Llama scores within a few points of the frontier models on the benchmark charts — why are we paying for API calls?" It is a fair question, and the honest answer requires separating two things the leaderboard summaries blur together: what a model knows, and what a model can do across fifty steps without losing the plot. Open-weight models have largely closed the first gap. The second is where SOC efficacy lives, and it is still open.
The gap, as measured by people who publish their work
On static knowledge and single-turn reasoning, parity is real. Strong open-weight releases have sat within low single digits of frontier models on MMLU-class benchmarks since 2024 — if your task is answering security-certification questions, model choice barely matters anymore.
Agentic benchmarks tell a different story. τ-bench (Sierra, 2024), which tests an agent completing multi-turn, tool-driven tasks under policy constraints, found the best proprietary model of the day, GPT-4o, solving only 35.2% of its airline-domain tasks — and the strongest open-weight models of that generation (Llama 3 70B, Mixtral 8x22B) trailing the proprietary frontier by a wide margin. On SWE-bench Verified, the long-horizon software-engineering benchmark, DeepSeek-R1 — the strongest open-weight reasoning model of early 2025 — reported 49.2%, while Claude 3.7 Sonnet, released weeks later, reported 62.3% (70.3% with a custom scaffold). Fixing bugs is not triaging incidents, but the underlying skill — sustained multi-step tool use against a large, messy state space — is the closest published proxy to L2 investigation we have.
Security-specific evaluation is thinner but pointing the same direction. CyberSOCEval, the open benchmark suite Meta and CrowdStrike released in September 2025 covering malware analysis and threat-intelligence reasoning, found that no current model comes close to saturating the tasks — and, notably, that reasoning-tuned models did not gain the improvements on cyber reasoning that the same tuning bought them in maths and coding. The frontier is not comfortably ahead on an absolute scale; it is ahead on a hill nobody has climbed.
One structural point explains most of what we see in production: errors compound. A model that is 97% reliable per step and one that is 90% reliable per step look similar in a demo. Across a 25-step investigation chain the first finishes clean roughly half the time; the second, under 7% of the time. Long-horizon benchmarks are brutal to mid-tier models for the same reason our investigation agent is.
Where the gap bites
Running identical agent workloads across model tiers, the failure pattern is consistent enough that we now predict it in scoping calls.
- L2 investigation chains. A typical Netgraph investigation runs 15–40 tool calls: graph traversals, log queries, enrichment lookups, hypothesis tests. Frontier models hold the thread — they remember that the third traversal contradicted the first hypothesis and adjust. The open-weight models we've run (70B-class, dense and MoE, through late 2025 vintages) reliably handle 5–10 steps, then start repeating queries they already ran, dropping constraints, or declaring victory on a hypothesis the evidence already killed. In one composite engagement, the same investigation playbook completed without human rescue 86% of the time on a frontier model and 41% on the customer's hosted 70B open-weight model. Same prompts, same tools, same graph.
- Root-cause synthesis. Writing the RCA that reconciles timeline, identity, asset and control-failure evidence into one causal account is the hardest text our agents produce. Frontier output needs light human editing; open-weight output at the 70B tier routinely produced plausible-sounding RCAs that our citation gate rejected for unsupported causal claims. This interacts badly with the faithfulness problem we wrote about in the explainability article: weaker models are not more honest, just more often wrong.
- Knowing when to stop. The most expensive failure mode is not wrong answers, it is confident premature closure. Our escalation-calibration checks (does the agent hand off when evidence is thin?) show frontier models escalating appropriately far more often. Under-escalation is the one you lose sleep over.
Where open-weight is genuinely fine
It would be self-serving to stop there, so: a large share of SOC LLM work is not long-horizon, and for that share, open-weight models are not a compromise — they are the correct engineering choice.
- Classification and extraction. Alert categorisation, entity extraction from unstructured logs, mapping report text to ATT&CK techniques. Single-shot, verifiable, high-volume. Open-weight models at the 8B–70B tier do this well, cheaply, on-prem, at latencies frontier APIs can't match.
- L1 first-pass with tight scaffolding. When the triage task is decomposed into small single-decision steps, each with forced graph citation and a constrained output schema, the open-weight gap narrows sharply. In composite production data, tight scaffolding recovered a hosted open-weight deployment from roughly 60% to roughly 85% of the frontier configuration's auto-close rate at equal audited accuracy — because every step small enough for the model is a step that can't silently derail. Caveat: these are operational medians across dissimilar environments, not a controlled eval.
- Summarisation and reporting drafts. Shift handovers, ticket summaries, first drafts of regulator notifications from graph-filled templates. The evidence is already assembled; the model is formatting, not investigating.
The sovereignty tension, and how we thread it
Here is the bind: the customers with the strictest model constraints are often the ones with the hardest investigations. Air-gapped manufacturing plants, defence-adjacent networks, and data-residency regimes that rule out cross-border API calls cannot use frontier models at all, whatever the efficacy delta. Telling them "run a worse SOC" is not an answer, so the platform is built to degrade gracefully instead.
| Deployment reality | Model plane | What changes operationally |
|---|---|---|
| Cloud-connected, no residency bar to APIs | Frontier API (customer's key) | Full agent autonomy within human-in-the-loop policy; highest auto-close rates. |
| Data must stay in-country / in-tenant | Hosted open-weight, customer infrastructure | Tight scaffolding profile: decomposed steps, stricter citation gates, lower auto-close thresholds, higher human sampling. |
| Air-gapped | On-prem open-weight + heuristic fallback | Agents recommend rather than close on complex chains; deterministic heuristics guarantee a floor if the model plane is down entirely. |
Three design rules make the middle and bottom rows workable. First, scaffolding does the reasoning the model can't be trusted to: the graph supplies ground truth, the planner supplies decomposition, and the model supplies judgement over small, checkable steps. Second, escalation paths are tuned per model tier — a weaker model is allowed to do less unsupervised, which costs analyst time but not accuracy. Third, the heuristic fallback means the platform's floor is a conventional, rules-driven SOC workflow, never a hallucinating one. We learned the hard way, at one air-gapped site in early 2026, not to let an under-provisioned quantised model limp along at 40-second token latencies — analysts trusted it less than no model at all, and they were right. Fallback thresholds are now automatic.
The trajectory, honestly
The gap is narrowing and pretending otherwise would age this article badly. Open-weight releases through 2025 and 2026 — the DeepSeek line, Llama 4, Qwen, GLM and Kimi families — have compressed the frontier lag on agentic benchmarks from roughly a year to something closer to months, and several open-weight labs now train explicitly for tool use and long-horizon tasks rather than chat. On current trend, the scaffolded-open-weight configuration we ship today will match today's frontier configuration well before the frontier stands still — which is the point: it won't. Our bet is architectural, not tribal. Because Netgraph is BYO-LLM, every improvement in open weights flows straight through to sovereign deployments without a platform change, and the day the efficacy delta stops paying for the API bill, our own recommendation flips with it. The graph, the traces, and the human-in-the-loop machinery are the durable investment. The model is a component on a swap path.
Key takeaways
- Knowledge benchmarks (MMLU-class) show near-parity; agentic benchmarks (τ-bench, SWE-bench Verified) still show a material open-weight–to-frontier gap — and per-step error compounds over investigation chains.
- CyberSOCEval (Meta + CrowdStrike, 2025): no model saturates SOC reasoning tasks, and reasoning-tuning gains from maths/coding did not transfer to cyber tasks.
- The gap bites on L2 investigation chains, root-cause synthesis, and escalation calibration; composite data showed 86% vs 41% unassisted playbook completion, frontier vs hosted 70B.
- Open-weight is the right choice for classification, extraction, and tightly scaffolded L1 first-pass — recovering ~85% of frontier auto-close rates at equal audited accuracy in composite deployments.
- Sovereignty is threaded, not ignored: BYO-LLM, per-tier scaffolding and escalation profiles, and a deterministic heuristic floor for air-gapped sites.
- The gap is closing on a months-not-years cadence; the durable investment is the substrate and controls, not the model.
Deployment anecdotes are anonymised composites of representative real outcomes; operational percentages are medians across engagements, not controlled evaluations. Published benchmarks cited: τ-bench (Sierra, 2024); SWE-bench Verified figures as reported by DeepSeek (R1, January 2025) and Anthropic (Claude 3.7 Sonnet, February 2025); CyberSOCEval (Meta and CrowdStrike, September 2025); CyberSecEval series (Meta).