Bash alone vs. typed tools for AI agents — adding typed tools to the shell did not raise the score

1. Executive summary

First, the terms. With typed tools, the model calls functions defined by a name, a description and a JSON schema, one at a time. Tools exposed by MCP (Model Context Protocol) servers are the typical example. A shell interface gives the agent a single shell such as bash and has it do all file handling, API calls and data processing through commands and scripts. With programmatic tool calling, the model writes a program that calls typed tools as functions; the program runs several tools in sequence inside a container and returns only the result. Anthropic and OpenAI both offer it as an API feature under this name: Anthropic's runs Python in a code execution container, and OpenAI's runs in an isolated JavaScript runtime. [5] [8]

The figures in this article are the values printed in the paper's tables. A difference in percentage points (pp) is the difference between two rates. Reduction percentages are derived values that we calculated by dividing values from the tables, and they are marked as such. The paper is research by Microsoft authors, supported by the Microsoft Copilot Studio team, and its introduction classifies Copilot Studio as a shell-based product built on the GitHub Copilot CLI and SDK. Here, "harness" means the agent execution loop that calls the model repeatedly and connects it to tools. We come back to this conflict of interest in section 10.

2. The five interfaces compared

The paper held the model, the task wording, the base prompt and the stop conditions fixed and changed only the tool interface. Only the Bash+Synthesis and programmatic tool calling conditions add instructions that explain how to use the interface. [1]

The third column is not from the paper; we added examples of setups in practice that look similar. The paper describes the five along two axes. A shell allows arbitrary code execution, so the agent can take several actions in one step, but execution and access have to be controlled. An approved tool list exposes only the functions the deployer allows, but calling them one at a time has a cost. Programmatic tool calling sits in between: a program chains, loops over and parallelizes the tool calls, but what can be done in the execution environment stays limited to the approved tool list. [1]

3. What was measured, and how

The score means something different on each benchmark. On TheAgentCompany it is a checkpoint completion rate weighted by difficulty. On APEX-Agents it is the average share of rubric criteria met, graded by an LLM-based grader. Pass rate is the share of tasks fully completed. Do not compare scores across the two benchmarks; compare interfaces only within the same benchmark. [1]

4. Result 1: score, tokens and cost

The table below reproduces the values from Table 1 of the paper. Tokens are the mean total tokens per task (input + output, including cache reads), and cost is the estimated inference cost per task. [1]

The Bash+Tool and Bash+Synthesis rows are in section 5. The paper's table gives 95% confidence intervals for the scores; they are roughly ±6 pp wide on TheAgentCompany and ±4 pp on APEX-Agents.

Three things stand out.

• The size of the score gap depended on the benchmark. On TheAgentCompany, bash alone scored 21.8–24.5 percentage points higher than typed tools, and the pass rate nearly doubled (24.1%→50.6%, 25.3%→47.7%). On APEX-Agents the gap was smaller, at 4.8–7.4 percentage points. The paper's interpretation is that flexible execution helps a lot with coding and moving between services, but helps less with domain reasoning.
• Bash used 19–72% fewer tokens. The largest difference was Opus-4.8 on TheAgentCompany, which dropped from 945k to 264k (a 72.1% reduction, by our calculation). The smallest was GPT-5.5 on the same benchmark, from 465k to 377k (an 18.9% reduction). On APEX-Agents, both models used about 46% fewer tokens. Cost moved in the same direction, but by less (15–69%, our derived values).
• Time did not consistently favor one interface. For Opus-4.8 on TheAgentCompany, bash was faster (5.1 min→3.0 min), but for GPT-5.5 typed tools alone were the fastest (1.8 min vs. 2.6 min). Where latency matters, look at tokens and time separately.

The paper's appendix recomputes these results as paired comparisons, matching the same task with the same model. On TheAgentCompany, the difference between bash alone and typed tools alone averages +24.6 pp over 348 pairs (95% confidence interval +20.6 to +29.0); on APEX-Agents it averages +6.2 pp over 960 pairs (+3.6 to +8.7). Neither interval includes 0. Broken down by domain, the order Bash ≥ Bash+Tool ≥ Tool-only held in every domain; the gap was largest in software engineering and smallest in corporate law. [1]

5. Result 2: adding tools to the shell

Attach MCP servers to a coding agent that already has a shell, and you get typed tools on top of a shell. That is the paper's Bash+Tool. The results are in the table of paired comparisons, matched by task and model. [1]

In all four comparisons, the confidence interval includes 0. In plain terms, the paper found no detectable score gain in its measurements. Tokens, on the other hand, mostly went up. For Opus-4.8 on TheAgentCompany, Bash used 264k and Bash+Tool 569k. The exception is Bash+Synthesis on the same benchmark (249k), the only token reduction the paper reports. [1]

The shell usage statistics offer a clue to why adding tools did not help. With typed tools present, the agent used the shell less, yet its remaining shell calls failed more often, and it was less likely to retry the same command right after a failure. On TheAgentCompany, Bash alone had a shell-call failure rate of 7.9% and an immediate retry-after-failure rate of 76.3%; for Bash+Tool, the figures were 10.8% and 58.7%. The paper sums it up: Bash+Tool "reduces shell use without improving the reliability of remaining shell calls." [1]

Bash+Synthesis, where the agent builds up its own tools, did not help either. The agent did use most of the tools it created (19 of 21 on TheAgentCompany, for Opus-4.8), but only about half were reused in later tasks (12 of 21), and there was no score difference between tasks that called synthesized tools and tasks that did not. Calls to reused tools did not succeed more often than freshly written code, either. [1]

6. Result 3: programmatic tool calling

Programmatic tool calling is an answer to a specific question: how do you keep the tool list but cut the cost of calling it? Anthropic's documentation says the approach lowers latency and token use by cutting round trips through the model, and that data can be filtered inside the program before it enters the model's context. It also gives figures from Anthropic's own measurements: performance on the BrowseComp and DeepSearchQA benchmarks rose 11% on average, and input tokens fell 24%. The benchmarks are third-party, but the vendor ran the measurements. [5] An independent August 2026 study, also cited by the paper, reported that on the BFCL v4 benchmark, programmatic calling matched or beat JSON calling for 11 of 14 models. [6]

This paper's measurements call for a narrower reading of the claim that programmatic tool calling is always better. Compared with typed tools alone, programmatic calling clearly saved tokens. In paired same-task, same-model comparisons, uncached input tokens fell by 97.6k per task on TheAgentCompany and by 45.4k on APEX-Agents. But the score rose only on TheAgentCompany (paired mean +3.5 pp, confidence interval +0.8 to +6.2); on APEX-Agents it was −1.5 pp, and the interval includes 0. And on both benchmarks, programmatic calling scored lower than the shell alone. The shell alone scored 21.2 percentage points higher on TheAgentCompany and 7.7 percentage points higher on APEX-Agents. [1]

Only three of the interfaces from Table 2 of the paper are shown. Success rate is the share of calls that reported success; repeated-call rate is the share of calls within a task that exactly repeated the same tool with the same arguments.

Programmatic calling had the lowest tool-call success rate on both benchmarks. On TheAgentCompany it was 65.2%, far below the roughly 90% of the other interfaces. The paper suggests this reflects the added difficulty of coordinating several calls and handling intermediate failures inside a generated program. In the failure analysis, too, programmatic calling had the highest rate of truncated final messages. Typed tools alone, by contrast, had the highest repeated-call rate, at 16.2%. The interpretation is that a shell or a program can bundle repeated work into one call, while typed tools send a separate call for every repetition. [1]

The paper's recommendation, as it stands: choose the shell alone when arbitrary execution can be isolated, and choose programmatic tool calling when security or compliance requires restricting the agent's actions to an approved tool list. Programmatic calling trails the shell on both score and cost efficiency (on raw cost alone, there is one combination where it is slightly cheaper: GPT-5.5 on TheAgentCompany, $0.54 vs. $0.56), but it offers a control advantage: it can confine what the agent is able to do to the approved tool list. [1]

7. Why the gap: failure types and shell usage patterns

The paper used an LLM-based grader to sort failed runs into 14 root causes. Failure types varied more by benchmark than by interface. On APEX-Agents, failures under all five interfaces clustered in missed requirements, misapplied domain rules and definitions, and numeric and formatting errors. On TheAgentCompany, the interfaces did differ. Typed tools alone most often failed by not finishing the deliverable or through environment and access errors, while the shell-based interfaces most often failed by declaring the task complete without verifying it. [1]

The paper also records how the shell was used, and the two benchmarks used it differently.

The Bash rows of Table 3 in the paper. The command count is the number of static commands, excluding embedded code and loop iterations.

On TheAgentCompany, the agent chained several commands into short calls, with heavy use of text utilities and HTTP clients (curl and others). On APEX-Agents, it ran twelve-line embedded programs, usually Python, inside the shell. The paper's summary: "Shell access can therefore support both utility-based workflows and file analysis." Syntax also differed by model. GPT-5.5 used heredocs for multi-line input far more often (82.7% vs. 16.8% for Opus-4.8, Bash alone on APEX-Agents), while Opus-4.8 chained commands with && and || more. [1]

To sum up, the reason the shell came out ahead in this paper is more specific than the model simply knowing bash well. Three behaviors came naturally with the shell interface: bundling repeated work into one call, processing intermediate results without putting them into the context, and fixing and retrying the same command right after a failure. Typed tools turn each of these into separate calls and separate model round trips. That is the mechanical cause of the token gap, and keeping the context short may be one path to the score gap. That last sentence is our inference; the paper does not establish causation.

8. How many MCP servers in the wild actually work

The typed-tools debate has another side: how usable are MCP servers in practice? A separate measurement published the same week speaks to it. Note first that it is a preprint by a single independent researcher. [2]

The study took a census of all 24,135 servers in the MCP Registry. From the servers distributed through npm that communicate over standard input/output (stdio), it then drew 400 at random with a published seed and tried to connect to each one once. The servers were probed as published, with no repairs and no credentials supplied. The results:

Half (51.2%) did not complete the initialize handshake. The main cause was not the commonly assumed lack of credentials (13.3%): servers that never started at all (37.5%) were nearly three times as common. In the paper's words, "roughly two in five published entries are inert." On a curated sample of 24 reference and popular servers, measured with the same instrument, 66.7% started. The author calls this 17.9-percentage-point gap "the size of the selection effect that curation introduces." Across the 2,766 tools exposed by the 195 servers that started, there were zero fatal JSON Schema violations, but the optional safety annotations that say whether a tool is read-only or destructive were missing on 58.8% of tools. Per server, it was all or nothing: a server annotated either all of its tools or none of them. [2]

This measurement's scope is also narrow. It covered only npm/stdio servers, which are 30.7% of the registry. Remote servers and PyPI and container packages are not in the sample. Each server got a single connection attempt with no retry, so transient outages counted as failures, and the author writes that 48.8% "is a lower bound on the fraction that could ever start." Still, the direction is clear. When you pick servers from the registry and attach them, the result depends on how carefully you vet them, and for more than half of the tools, the tool itself does not tell you whether it is safe. [2]

Read together with the results in section 7, attaching typed tools carries two separate costs: the cost of attaching them without the score going up (this paper's measurement), and the cost of choosing and vetting the ones worth attaching (the registry measurement). We covered security controls for MCP separately in our coding agent security report.

9. Checklist to apply today

• Count the attached typed tools that the shell could also handle. These are tools that curl, jq or Python can replace, such as file reads and writes, HTTP API calls and text search. The tool lists in this paper were made up of this kind of tool, and adding the whole list brought no score gain. The effect of each kind of tool was not measured separately.
• Keep only the tools the shell cannot reach. That means internal systems whose auth tokens you cannot hand to the agent, execution environments without a shell, and work where only approved actions may be allowed. These cases are outside the paper's experimental scope, so do not apply its results to them.
• If you decide to give the agent a shell, set up isolation first. The paper's shell also had file system restrictions, environment variables removed and execution limits, and during development there was reward hacking in which the agent read grading scripts. We covered isolation approaches in our sandbox report.
• If policy requires an approved list, try programmatic tool calling first. It clearly used fewer tokens than typed tools alone. But its call success rate was the lowest, so check the failure handling inside the programs in your logs.
• Measure score, tokens and time separately. For GPT-5.5, typed tools alone were the fastest. Do not assume that fewer tokens means lower latency.
• Check three things when picking MCP servers from the registry. Does the server actually start without credentials? Do its tools carry safety annotations (readOnlyHint, destructiveHint)? Is the tool something the shell could replace?
• Watch for tasks declared complete without verification. This was the most common failure type for shell-based interfaces. Put a step that actually checks the deliverable into the prompt or the harness.

10. What this is not yet

• This is vendor-affiliated research. All authors are at Microsoft, the work was supported by the Copilot Studio team, and the paper's own introduction classifies Copilot Studio as a shell-based product built on the GitHub Copilot CLI and SDK. The experimental harness is also the GitHub Copilot SDK. The results align with the company's product direction. That does not mean the data is wrong; it means the results should be read with that fact in mind until they are reproduced independently.
• It is a preprint, not yet peer-reviewed. Published September 10, 2026, in the AAAI 2027 format.
• Only two models. Opus-4.8 and GPT-5.5, at default reasoning effort. The paper itself notes that "these recommendations may not carry over to later releases in either family." Whether the shell alone gives the same result with smaller or open models was not measured.
• The 60 typed tools for TheAgentCompany were written by the authors. The benchmark had no official list. The quality of the tool descriptions and schemas can affect how the typed-tool conditions perform, and the size of that effect was not measured.
• On APEX-Agents, the shell was an MCP server. The code execution server that the benchmark provides served as the shell. So this paper compares "one shell vs. a typed tool list," not "MCP vs. bash." Whether the shell was delivered over MCP was not an experimental variable.
• The absolute scores cannot be compared with public leaderboards. The APEX-Agents leaderboard uses maximum reasoning effort and the latest server configuration; this paper used default reasoning effort and the configuration as of June 5. For TheAgentCompany, the authors used a version in which they had fixed defects in the coworker simulation.
• Costs are list-price estimates. They are calculated from GitHub Copilot list prices, not actual bills or other providers' prices.
• The task-complexity analysis is weak evidence. The paper itself qualifies it: for programmatic calling, the confidence intervals include 0 in most complexity bins, and the number of typed-tool calls is an imperfect proxy for complexity.
• The causal explanation in the last paragraph of section 7 is our inference. The paper explains the mechanical cause of the token gap, but it does not establish that this is the cause of the score gap.
• The MCP Registry measurement is a single-author preprint with a narrow scope. It covers only npm/stdio servers (30.7% of the registry), each probed once with no retry. The author also describes 48.8% as a lower bound. The two papers differ in authors, methods and questions; putting them side by side is our choice.
• We did not run our own measurements. This article is a summary based on checking the LaTeX sources of the two preprints and the vendors' documentation.

11. Sources

All checked on 2026-09-15. For [1] and [2], we downloaded the LaTeX sources from arXiv and checked the tables and appendices directly. Product documentation and vendor blogs are the providers' own descriptions, not independent experiments. [7] is a secondary source that shows the context of the debate. [8] is placed before [7] in the table below.

Editorial standard: we carried over as figures only the values printed in the paper's tables, and marked the reduction percentages we calculated as derived values. We placed the conflict of interest of a vendor-affiliated study, and the setup fact that the shell was delivered through an MCP server, where readers see them before the conclusions.