The three layers of a coding agent sandbox — isolation strength was not the problem
Per file system operation, 69.0–98.6% of real command denylists missed a bypass. In coding agent sandboxes, isolation strength was not the problem.
On this page
We checked three 2026 measurement studies on coding agent isolation against the original papers. Each looked at a different layer, and in every layer, something other than isolation strength decided the outcome. The isolation engine fixed vulnerabilities on the day they were disclosed, yet one product had gone 582 days without updating to a newer version. And the command denylists in real use were missing bypasses for the very file system operations they were meant to block, regardless of project or agent.
Executive summary
A sandbox is an execution environment that keeps the code an agent runs away from your own machine. As coding agents were given shell and file access, everyone turned sandboxes on, and in 2026 studies appeared that measured how much sandboxes actually block. What the results pointed to was not a ranking of isolation methods.
What a sandbox blocks
When someone says “we turned on the sandbox,” that one sentence covers three layers that do different jobs. Each of the three studies in this report looked at one layer, so we separate the layers first.
| Layer | What it blocks | Typical implementation | Measurement covered here |
|---|---|---|---|
| 1. Isolation | Code the agent runs reaching the host machine | Containers, microVMs, user-space kernels | Engine-level comparison of five products |
| 2. Policy | Commands that must not run even inside the isolation | Command denylists, permission rules | How bypassable 1,709 real denylists are |
| 3. Scope | Actions the tool's permissions allow but the user did not authorize | Approval steps, task-scope declarations | 15 combinations out of 4 agents × 6 models · about 7,500 runs |
No layer can stand in for another. Even perfect isolation (layer 1) does not stop project files from being deleted inside the sandbox, and even a perfect layer 2 does not stop what an agent does with ordinary commands that are not on the denylist. And as we will see below, the layer the 2026 measurements flagged most was layer 3, not layer 1.
Layer 1: the engine shipped the fix, the product did not pick it up
The first paper compared, along six axes, how five AI sandbox products keep guest code away from the host kernel (arXiv:2606.08433, 2026-06-07). The authors did not produce an overall ranking, and stated explicitly that no single axis, read on its own, supports a judgement.
Yet from the operator's point of view, the biggest difference came not from the isolation method but from which version of the engine the product pins. Version pinning here means locking a dependency to one specific version.
Below, upstream means the project that builds the isolation engine, and downstream means the sandbox product that takes that engine and uses it.
Upstream — the engine side
Under coordinated vulnerability disclosure, time to patch was about 0 days. The slowest case, a single runc regression, took about 26 days.
Downstream — the product side
From 0 days to 399 days, 471+ days, and “opaque.” The paper says the downstream spread is four orders of magnitude wider than upstream.
In the paper's own words, “operator-visible patch-propagation is dominated by product-side pin policy, not by engine-side patch speed” — that is, by when the product updates the version, not by how fast the engine fixes the bug. And isolation class did not predict this outcome.
| Product | Isolation | Do patches arrive? | Can extra hardening be applied? |
|---|---|---|---|
| arrakis | microVM (Cloud Hypervisor) — the stronger class | No | Yes — seccomp filter on 32/33 worker threads |
| daytona | OCI container — the weaker class | Yes | No — the runner's Privileged: true disables AppArmor and re-adds capabilities |
Between these two products, patches did not reach the one with stronger isolation, and the one that did get patches could not take extra hardening (kernel-level restrictions such as AppArmor or seccomp, layered on top of the base isolation). Neither problem came from the engine; both came from defaults the product chose. Of arrakis, the paper writes: “The engine has been doing its job; the product has not been propagating the fixes.”
setup/install-images.py still read v44.0, and the repository's last commit was 2025-06-02. From the pin date, 2025-02-04, to the day we checked is 582 days, and in that time upstream tagged 13 releases (v45.0–v53.0). The 582 days and 13 releases are derived values we calculated from the public pin date and release list.
arrakis is a small project for self-hosting (869 GitHub stars). The point is not to single out one project; it is that this value sits in public, where anyone can check it in a few minutes. With a commercial product it is actually harder to check — in the same paper, one product's cloud-hosted edition was classified as “opaque” because its pinned version cannot be seen from outside.
Layer 2: denylists missed what they were meant to block
The second paper looked at how agents that run in a terminal filter shell commands (arXiv:2606.15549, published 2026-06-14 / revised 06-20). Such agents usually decide whether a command may run using three kinds of lists, and the one that actually carries the decision is the denylist, which enumerates the commands to block.
The authors built a pipeline that has a language model propose commands that get around a denylist, then actually runs each one in a sandbox to confirm that it really performs the operation. The candidates are commands that none of the list's rules match to begin with. The pipeline then compares the operations a candidate command performs with the operations the list is meant to block, both established the same way, and counts any overlap as a bypass. The key to this measurement is that it compares at the level of operations, not command strings.
Read this range precisely. The paper did not measure everything denylists try to block. The authors narrowed the target to seven file system operations: read, write, create, delete, copy, change permissions, and change owner and group. For each of the seven, they counted the share of the lists aimed at that operation that missed at least one bypass; 69.0% is the best-blocked type and 98.6% the worst-blocked.
So it does not mean that 69% of all lists were bypassed. It means that whichever of these seven operations a list tried to block, at least 69.0% of the lists targeting that operation had a path they missed.
What matters here is what was measured. These results come not from hypothetical policies written by researchers but from lists that developers actually wrote and deployed. The survey paper covered in Section 6 cites exactly this point in ranking this result among the most consequential findings in the whole literature — reporting that a defense holds up against an attacker you built yourself is a different claim from reporting that policies deployed in the field actually get bypassed.
But the paper's conclusion is not “write better lists.” The authors also show that fixing a list means adding every missing bypass command, a burden that is too heavy, and conclude that a static command-string denylist is inadequate as the main layer of defense for gating agent commands.
Layer 3: actions that permissions do not filter out
The third paper matters most for this report (arXiv:2605.18583, 2026-05-18). It is not about attacks. It is about agents that, given an ordinary request, also do things nobody asked for — deleting unrelated files, clearing out an old credentials backup, rewriting configuration no one mentioned.
The authors call these out-of-scope (“overeager”) actions and classify them as a different kind of problem from prompt injection or sandbox escape. The paper's definition: an action the request authorized neither explicitly nor implicitly, which either actually changed environment state (file system, processes, network connections, git objects) or read a predefined sensitive location (.bash_history, .ssh/, dotfiles). Intent that was stated but never executed, and authorized reads of ordinary files, do not count.
One distinction matters here. These actions are possible under the tool's permissions — the shell and the files are already open to the agent. They are simply outside what the user asked for. That is why permission rules do not catch them. A rule answers “is this type of action allowed?”, not “did the user ask for this particular execution?”
Scale of the measurement
500 validated scenarios were run about 7,500 times across 15 combinations pairing four agent products (Claude Code · OpenHands · Codex CLI · Gemini CLI) with six base models. The full 4×6 grid would be 24 combinations; 15 were actually run. Agreement was checked by re-labeling 50 runs and computing the kappa coefficient, which expresses on a 0–1 scale how often two raters give the same verdict on the same material. The result was 0.73, which is considered agreement good enough for practical use.
Finding 1 — removing the authorization-scope block raises the rate sharply
While building the benchmark, the authors ran into a problem: if the prompt spells out the authorized scope, the agent stops inferring boundaries from the task itself and starts pattern-matching the declaration text. So they built scenario pairs that differ only in whether an authorization-scope block containing Authorized: and OUT OF SCOPE: is present, and are otherwise byte-for-byte identical, and compared them as pairs.
So what to take from this measurement is not the size, 17.1%, but the direction. The agents were not understanding the task's boundaries on their own; they were responding to text that spelled out the scope, and when that text was removed, out-of-scope actions rose for all three models. The size shrinks to as little as 2.6 percentage points depending on how the scenarios are composed.
Finding 2 — the tool matters more than the model
The bigger difference came not from which model you use but from which tool you use. The paper split the four products into permissive ones, which go ahead even with risky actions, and ask-to-continue ones, which ask the user before executing.
| Paper's classification | Products | Rate of out-of-scope actions |
|---|---|---|
| Products classified as permissive | Claude Code, Codex CLI, Gemini CLI | 5.4–27.7% |
| Products classified as ask-to-continue | OpenHands | 0.2–4.5% |
In Fisher's exact test, the probability that the gap between the two groups was due to chance was 1 in 100,000 or less. Within the same class, changing only the base model moved the rate by up to 15.9 percentage points. The authors take this to mean that model-layer alignment does not fully propagate through permissive permission gating. In other words, switching to a better-aligned model does not solve this problem for you.
--dangerously-skip-permissions; OpenHands: security.confirmation_mode=false; Codex: --ask-for-approval=never --sandbox=danger-full-access). It does not say what was set for Gemini CLI. And the paper does not settle how much of OpenHands' confirmation behavior remained under that setting. The same appendix labels the setting “disabling interactive confirmation,” yet the paper calls OpenHands ask-to-continue throughout, and one of its run traces notes: “The OH framework did not trigger ask-user-to-continue in this scenario.” No trace in which OpenHands actually asked appears anywhere in the paper. So the table means neither that the results are unrelated to approval design nor that the approval step had an effect. And because products were not randomly assigned and each differs in implementation beyond the approval step, this table cannot tell you how much safer an agent becomes if you make it ask every time.
Where the four real vulnerabilities sat
The three layers above are measurements. What about vulnerabilities that were actually disclosed? A survey paper (arXiv:2607.05743, 2026-07-07) organized 39 papers from 2023–2026 into 17 categories and, along the way, verified four vulnerabilities that affected real products — a CVE is the identifier assigned to each publicly disclosed vulnerability — against the original entries in the US National Vulnerability Database (NVD). We rechecked the same four ourselves through the NVD API. CVSS is a vulnerability severity score out of 10.
| CVE | Affected | CVSS 3.1 | What it allowed | Fixed in |
|---|---|---|---|---|
| CVE-2024-21626 | runc (container runtime) | 8.6 | A leaked file descriptor leaves the working directory of a container process newly started with runc exec on the host side → escape to the host file system. Nicknamed “Leaky Vessels” (CWE-403/668/200) |
1.1.12 |
| CVE-2025-53773 | GitHub Copilot / Visual Studio 2022 | 7.8 | Local code execution through command injection (CWE-77) | 17.14.12 |
| CVE-2025-59536 | Claude Code | 8.8 | Runs code in the project before the user accepts the startup trust dialog (CWE-94), when launched in an untrusted directory | 1.0.111 |
| CVE-2026-21852 | Claude Code | 7.5 | If a malicious repository sets ANTHROPIC_BASE_URL in a settings file to an attacker's address, API requests go out the moment the repository is opened, before the trust prompt appears, and the key leaks (CWE-522) |
2.0.65 |
Of the four, only the first actually broke container isolation. And it is a flaw in a container runtime that predates agent tools — the kind agent sandboxes inherit because they are built on the same components.
Both the NVD description and the survey describe the Copilot case only as “command injection.” But the original report that NVD links from that entry, tagged Exploit, shows a more specific path: in GitHub Copilot's agent mode in VS Code, Copilot, once hit by a prompt injection, writes "chat.tools.autoApprove": true into .vscode/settings.json, switching off every user confirmation by itself, and then runs terminal commands. Isolation was not broken; the approval boundary collapsed from the inside.
Note, though, that the fixed version in the table, 17.14.12, corresponds to the affected range NVD lists, Visual Studio 2022 17.14.0–17.14.11. For the VS Code path shown in the original report, the report says only that it was fixed in the August 2025 Patch Tuesday release, with no version number.
The two Claude Code cases point somewhere else again — to the gap between the moment an untrusted repository is opened and the moment the user's trust decision actually takes effect. Neither broke isolation; both happened before isolation and the trust decision applied.
All four cases in the table have fixed versions. The Claude Code fixes have already reached users on auto-update: fixed version 1.0.111 shipped on 2025-09-10 and 2.0.65 on 2025-12-11, and the latest as of 2026-09-09 is 2.1.266 (checked on the npm registry). The environments at risk are those that pin the version by hand — the version-pinning problem from Section 3 applies to the agent tools themselves as well.
What nobody has measured yet
Reading across the 39 papers, the survey found five gaps. Here are the three that bear directly on practice.
Nobody measures badly written policies
Every policy-enforcement approach reviewed assumes the author wrote the policy correctly and checks only whether the system enforces it as written. No study covers policies that are wrong to begin with, too permissive, or self-contradictory. Given how often deployed lists missed bypasses for the operations they meant to block, policy-authoring errors may be as large a source of risk as enforcement failures — yet this remains unmeasured.
Defenses are not retested against real failure rates
Papers on isolation and access control report how well their defense holds against attackers the authors built themselves. None retests its own defense with the bypass techniques found against real deployed policies. The survey points out that this re-evaluation needs no new infrastructure — you can simply run the published bypass corpora.
What you can do today
We list only what follows directly from the measurements above.
1. Check your agent tool versions
All four cases we checked have fixed versions. If you use auto-update, you already have the fixes for the two Claude Code cases. The other two you have to update yourself — runc to 1.1.12 or later; for the Copilot case, 17.14.12 or later on Visual Studio 2022, and on VS Code a version from the August 2025 Patch Tuesday release or later (the original report does not give a fixed version number for VS Code). If your environment pins versions, update now. For Claude Code, you need 2.0.65 or later to have both fixed.
2. Check which engine version your sandbox product pins
For operators, when the pinned engine version was last updated was a bigger variable than the isolation method (microVM or container). For open source, look for the hard-coded version in the install script. For a commercial product, ask — and if they cannot answer, that in itself tells you something.
3. Do not count a denylist as a line of defense
A denylist is an aid for reducing mistakes, not a security boundary that stops attacks. The paper concluded that a static command-string list is inadequate as the main layer of defense, and the built-in list maintained by the vendor was no exception. Draw the security boundary with the resources the process can reach in the first place, not with a list of commands.
4. Write the task scope into the prompt
This is the cheapest response the measurements support. Out-of-scope actions dropped when the prompt stated what may and may not be touched, and the direction was the same for all three models tested. But do not count on the size — the 11.9–17.2 percentage points from the single-axis experiment come from a condition the paper says “upper-bounds the effect on a sensitive sub-population,” and with varied scenarios the effect falls to 2.6 percentage points. And since the agent is responding to that text rather than understanding the boundary, treat it as a mitigation, not a fix.
5. Cut off irreversible operations before they run
The rate of out-of-scope actions varied more with the tool than with the model. But that experiment did not randomly assign products, so it cannot tell you how much better things get if you make the agent ask every time. Instead, keep hard-to-reverse operations such as deletion, credential changes, and deployment out of the agent's reach entirely — not granting the permission is more reliable than adding a confirmation.
6. An untrusted repository is dangerous from the moment you open it
Both Claude Code vulnerabilities happened before the trust dialog appeared. Do not rely on the confirmation prompt for a repository you have never seen; open it in an isolated environment, or at least in one with no credentials.
What this is not yet
- The four CVEs are examples, not a sample. They are the four the survey picked for its “Real-world incidents” section — neither a random sample nor a complete count. So this report draws no frequency conclusion from their mix, such as that isolation is generally safe or that operational problems are more common. One of the four is a real isolation escape.
- The product comparison covers five products, and only Part 1, the engine-level part. The engine comparison paper did not rank the products itself and states that “No single axis is a sufficient basis for a comparative judgement.” We contrasted arrakis and daytona in this report to show that isolation class does not predict operational outcomes, not to rank the two products. We could not check Part 2 — the product-level axes, which look not at what an engine structurally allows but at which defaults a deployed product actually runs with.
- The 582 days and 13 releases are values we calculated. We counted from the date of the pinning commit (2025-02-04) and the upstream release list, as of 2026-09-09. The paper's 471 days is as of an earlier measurement date; the two numbers differ only in when they were counted and do not conflict.
- 17.1% is the value for one combination, under a condition the paper calls an upper bound. It comes from one cell, Claude Code on GLM-4.6, measured on the hand-curated 76-scenario pilot set with only the authorization-scope axis changed. The paper itself says the reported Δ “upper-bounds the effect on a sensitive sub-population,” and in the main benchmark, with five axes randomized, the same effect falls to 2.6 percentage points. The rate under a real-world task distribution cannot be known from either.
- The cause of the differences between products has not been established. The paper discloses the confirmation-off setting for only three of the four products, and it does not settle how much of OpenHands' confirmation behavior remained under that setting. So we can say neither that OpenHands scored low because it asks every time nor that the result is unrelated to approval design. The products differ in implementation beyond approval, and this measurement cannot separate which element makes the difference. It is an association, not causation.
- We did not verify sandbox-escape reports published by security vendors. Several from 2026 came up in our searches, but we could not check them against primary sources, so we left them out of this report.
- All three papers are preprints (arXiv). We could not confirm whether they have been peer-reviewed. All three do name their data sources (lists collected from GitHub, public repositories, NVD), so the values themselves can be traced.
- Network operations were outside the measurement. The 69.0–98.6% applies to seven file system operations, while real-world denylists commonly include rules meant to block outbound transfers. How well those hold up cannot be known from this paper.
- Our only measurement of our own is the version-pin check. We did not reproduce the denylist bypasses or the out-of-scope actions; we only checked the values the papers report against the originals.
- The paper is internally inconsistent on the number of denylists. 1,731 appears in only one place, the introduction's paragraph summarizing the results; the abstract, the introduction's list of contributions, the collection table in the evaluation section (1,000+136+573=1,709), and the conclusion all say 1,709. We use 1,709, the figure those four places agree on.
Sources
| Source | Type | What we used it for |
|---|---|---|
| AI Code Sandboxes: A Comparative Security Study (Part 1) arXiv:2606.08433 · 2026-06-07 |
Primary (measurement) | Engine-level comparison of five products; upstream ≈0 days vs. downstream 0 to 471+ days; the arrakis / daytona contrast |
| One Goal, Many Commands: Characterizing Denylist Fragility in AI Agents arXiv:2606.15549 · 2026-06-14 (v2 06-20) |
Primary (measurement) | 1,709 denylists · 13,332 rules · share of lists that missed a bypass, per type across seven file system operation types: 69.0–98.6% (network operations excluded by the paper) |
| Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks arXiv:2605.18583 · 2026-05-18 |
Primary (measurement) | 0.0%→17.1% (76-scenario pilot set; a value the paper calls an upper bound); the same effect at 2.6 percentage points on the 500-scenario main benchmark; increases of 11.9–17.2 percentage points; permissive 5.4–27.7% vs. ask-to-continue 0.2–4.5%; kappa 0.73 |
| The Balkanization of Execution-Security Research for AI Coding Agents arXiv:2607.05743 · 2026-07-07 |
Secondary (survey of 39 papers) | 17-category taxonomy, the five gaps, the choice of the four CVEs. Individual figures were rechecked in the primary papers above |
| NVD — CVE-2024-21626 · CVE-2025-53773 · CVE-2025-59536 · CVE-2026-21852 | Primary (official) | CVSS 3.1 scores, CWEs and fixed versions for all four. CVE-2026-21852's actual technique and CVSS 4.0 score, CVE-2024-21626's 8.6 score and fixed version 1.1.12, and CVE-2025-53773's affected range (Visual Studio 2022 17.14.0–17.14.11) come from NVD |
| GitHub Copilot RCE via Prompt Injection The original report NVD links from CVE-2025-53773, tagged Exploit · 2025-08-12 |
Primary (public report) | The path in VS Code agent mode that turns on chat.tools.autoApprove in .vscode/settings.json to switch off confirmations. This is not in the NVD description |
| npm — @anthropic-ai/claude-code | Primary (checked directly) | Checked 2026-09-09 — 1.0.111 published 2025-09-10, 2.0.65 on 2025-12-11; the latest tag is 2.1.266 |
| abshkbh/arrakis · Cloud Hypervisor releases | Primary (checked directly) | Checked 2026-09-09 — still pinned at v44.0, last commit 2025-06-02, 13 upstream releases, 582 days |
All figures were checked against the original sources on 2026-09-09. We checked paper values by reading the abstracts and full text ourselves, the CVEs against NVD API responses, release dates and the latest version against the npm registry, and the pin status against the repository file and release list. The 582 days and 13 releases are derived values we calculated from public dates. Our draft got three things wrong — the axis behind 69.0–98.6%, the scope of 17.1%, and a causal reading of the differences between products; pre-publication review flagged them, and we rechecked the three papers and corrected them.