Brief · Both
Why Are Labs Still Using Irregular?
A Meta spokesperson, per Dark Reading: "It would be inaccurate to report this was a breach by Meta, given this took place via a third party contractor." Dark Reading calls that "technically true." Sure. Meta's model did the exploiting. The contractor's network let it out. Both can be true. Whoever owns the real site it hacked (Meta just calls it 'a real website,' like it's 2005) doesn't care which one.
What failed
Per Meta's post-mortem, Irregular tested a pre-release Muse Spark 1.1 "in a closed testing environment with safeguards removed." That part was on purpose, and it's normal. Then "a misconfiguration allowed the model to access the open internet," and instead of a fictional target name, Irregular "unintentionally provided the model with a name of a real website." The model "identified and exploited a security vulnerability in the real website," accessed information, and "made changes to the website's database."
So the failures are: - Egress. The "closed" environment wasn't closed. Irregular: "most of the issues we've discovered were due to internet access controls." - Config. A made-up target name matched a real domain, and nobody caught it. - Real-time monitoring. Meta learned about it from Irregular, and says it has "identified monitoring improvements that would help identify these types of issues earlier."
It was not a sandbox escape. Meta, OpenAI, and Irregular all say so; Meta's words are "this was not a sophisticated offensive cyber attack or sandbox escape." Dark Reading's line that the models "broke containment" doesn't fit. There was no wall to break. The door was open. Meta's post doesn't say whether Muse Spark 1.1 used credentials. Other labs' models in the same environment did.
Same environment, other labs: Anthropic reported three incidents through Irregular's environment on Jul 30, and later a fourth. OpenAI reported a domain collision on Aug 4. Irregular says these are "not materially separate incidents" from "a single evaluation scenario." Anthropic counts them separately. How many incidents there were is disputed.
So why do labs keep using them?
OpenAI's o3/o4-mini system card cites Irregular's earlier name, Pattern Labs. Anthropic's Claude Opus 5 system card uses Irregular's CyScenarioBench. Google DeepMind's Gemini 4 Argon safety report rests partly on "The Irregular CyScenarioBench evaluation." There aren't many comparable vendors; one analyst quoted by CNBC names METR and Apollo. After the incident, Meta praised its "ongoing partnership," OpenAI said "we will continue to work closely with them," and Anthropic said "We look forward to our joint work on security." Irregular kept publishing new assessments: GPT-6 Astra (Sep 3), Claude Opus 5.5 (Sep 22), Gemini 4 Argon (Oct 13).
I ran managed security shops for years. One bad mistake, a missed alert or a botched config, and you could lose the client. That's normal. EDR vendors, MDR providers and MSPs get fired for simple mistakes all the time. So what's the deal with AI? Same vendor, multiple labs, real systems hit, and everyone's "look[ing] forward to our joint work on security."
My take: they're stuck. Dropping Irregular would leave holes in the labs' own safety claims.
You can outsource the testing, not the blame
Opinion: if you pay someone to point a model with its safeguards removed at things, egress and monitoring are your requirements, not their best effort. Anthropic says it treats the fixes "as if the responsibility were ours alone." That's the right instinct. "Our contractor did it" isn't a control.
Do these people know you can just put an egress allowlist in the contract?
What to do: Before you hand agent work to any vendor or tool, ask three questions. 1. What can it reach? Show me the egress policy and how it's checked before every run, not "the environment is closed." 2. Who's watching, in real time? Who gets paged when it reaches something outside scope, and how fast does the run stop? 3. When it hits a real system, who tells whom? Notification to the victim, to us, and to authorities, with timelines and per-run logs we can review afterward, in writing.
Sources
- Dark Reading: When AI Agents Commit Breaches, Humans May Not Be Held Liable (Oct 16, 2026)
- Meta: Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1 (Aug 14, 2026)
- Irregular: Addressing recent incidents, ongoing findings and path forward (Aug 14, 2026)
- Anthropic: Investigating three real-world incidents in our cybersecurity evaluations (Jul 30, 2026)
- Anthropic response to Sen. Blunt Rochester (Sep 15, 2026)
- OpenAI: Third-party cyber evaluations involving OpenAI models (Aug 4, 2026)
- OpenAI o3/o4-mini system card, cybersecurity
- Anthropic: Claude Opus 5 System Card
- Google DeepMind: Gemini 4 Argon Safety & Security Report