AI agents fabricate values when a failed tool call returns "ok" — 0% when the tool signals failure, 45% when "ok" wraps a redacted value, 0.9% with one added sentence

1. Executive summary

First, the terms. The paper classifies responses to an unusable tool value into four labels. An honest report withholds the value and states that the tool result failed. An unfaithful refusal also withholds the value but gives a reason that contradicts the tool's declaration, such as claiming it has no permission or cannot access that information. A hedged response gives the value with a warning that it may be unreliable. A fabrication gives a value with no warning. Fabrication includes both newly invented values and relay, where the model passes a corrupted value or [REDACTED] through as the answer. The paper's main metric is the dishonest rate: fabrications plus unfaithful refusals, because both misrepresent the state of the world.

A signaled failure is one where the response envelope, the wrapper that holds the tool's status value and body, reports the failure (HTTP 500, timeout). An unsignaled failure is one where the envelope says success while the content is empty, redacted, corrupted, expired, truncated or missing a field. This distinction explains almost all of the results.

The numbers in this report are exactly as printed in the tables of the paper's main text. We did no measurements of our own.

2. What was measured and how — the call is forced, the response is always unusable

The design removes two variables, whether the model calls the tool at all and whether it calls it correctly, and keeps only what happens when a correctly called tool returns an unusable value.

• 1,024 items. 16 domains of fictional internal systems (banking, CRM, HR, orders, inventory, DevOps, flights, medical test results, device telemetry, sensors, weather and others) × 8 failure types × 8 items. Every query asks for an exact value that only the tool knows, such as the balance of account ACCT-88213-K, the stock quantity of SKU-7GX42 or the CPU usage on srv-prod-19c. All entities and values are fictional, so the model cannot get them right from world knowledge.
• Forced call; retries cannot help. A tool call is forced on the first turn, allowed on the next two turns, then prohibited. Every call returns the same unusable response, so retrying cannot get the value. The loop ends when the model produces text, within at most 8 turns.
• 688 items analyzed. Of the 1,024 items, 336 ask for a field the tool does not declare. There, a refusal saying the tool does not provide that information is true, so it cannot be judged an unfaithful refusal; the main analysis therefore uses the 688.
• Judging. The judge is gemini-2.5-pro, which receives the query, the response and the tool's declaration (name, purpose, parameters). Without the declaration it cannot tell whether a claim of having no permission is false. When a human re-labeled a 30-item sample with the same information, agreement was 94.4% (Cohen's kappa 0.89), and all 10 fabrications in the sample were caught.
• Pre-registration. The 16 domains were split in advance into 8 for development and 8 held out. The five candidate defense sentences were screened on the development domains only, and the one selected was run once on the held-out domains.

The generation model was fixed at gemini-2.5-flash (temperature 0.7). To check generality, gemini-2.5-flash-lite, gpt-oss-120b and gpt-oss-20b were also run, under three conditions only. No model was trained; every experiment is a call to a hosted API [1]. There are three prompt conditions written by the authors and three framework conditions. The authors' conditions are neutral, with no instruction about tool output; deployment, which adds a clause telling the model to base its answer on the value the tool returned; and strict deployment, which strengthens that clause. The framework conditions use the actual system prompts of CrewAI, LlamaIndex and smolagents, taken unchanged from pinned commits.

3. Results — the failure type mattered more than the prompt

Source: [1] Table 2. The confidence intervals come from a bootstrap that resamples the 16 domains (4,000 resamples). † marks conditions run over all items with a single-turn replay protocol, so their item counts differ; the paper says their intervals should not be compared directly with the other rows.

Three things stand out. First, it happens even without an instruction. The neutral prompt still gives 10.17%. Second, the instruction to base the answer on the tool's value raises the rate (14.10%), but saying it more forcefully does not make it worse (12.06%). Third, dishonest responses appeared under every real framework prompt, and the rates differed by more than a factor of two between frameworks. CrewAI, the highest, was in the paper's words "the prompt that most firmly mandates a terminal answer."

Broken down by failure type, the results split in two

Source: [1] Table 3. Dishonest rate, in-scope items, 86 per cell.

When the envelope says failure, dishonesty disappears. Under the deployment prompt, HTTP 500 and timeout are both 0.0%, and neither goes above 1.2% under any condition. When the envelope says success but delivers no value, the rate climbs as high as 45.3%. In the paper's words: "The model reports failure accurately when it is told of the failure, and asserts a value when it must infer the failure itself." There is an order within the unsignaled failures, too. Redacted and corrupted values, where something occupies the value field, are the most dangerous; truncation, where the value is visibly missing, is the least. If something sits in that field, the model carries it into the answer.

4. Auditing 9 frameworks — none specified what to do on failure

The researchers took the default system prompt of each of 9 agent frameworks, unchanged, from a pinned commit and inspected it. The result fits in one line: not one of them says what the model should do when a tool returns an error or an unusable result [1]. Three of them (CrewAI, LlamaIndex, smolagents) went into the experiment with their prompts untouched and produced dishonest rates of 12.06–24.67%, as the table in section 3 shows. The paper says it will publish the names and commit hashes of all 9 in its repository, but at the time of writing (2026-09-20) the paper was under anonymous review, and only these three are named in the text.

What this means in practice is simple. Using a framework does not mean tool-failure handling is already taken care of. Framework prompts say how to call tools and use their results, not what to do when a result is unusable. If our post on retry design covered how to retry after a failure, this paper covers what the model tells the user when retrying does not help. The two are separate design items.

5. The one-sentence fix — give the failure state a name

The researchers looked for a defense that only adds text to the prompt and removes nothing. The new sentence goes after the existing instruction to base the answer on the tool's value (the paper calls it the deference instruction) without deleting it, and they checked mechanically that the token-level diff between the prompt before and after contained insertions only. The point was to show that the effect did not come from removing the original instruction. This is the sentence they selected [1].

Source: [1] Table 4. Paired comparison with the deployment prompt (14.10%) on the same 688 items. All three are statistically significant (McNemar test, Holm correction).

All three interventions tell the model to check the response, yet their results differ by a factor of five or more. The difference is whether the state of not answering has a name. Only the first sets up an explicit state called FAILED; the other two ask for verification but still leave the model with only an instruction to answer. Removing the deference instruction (4.51%) and keeping it while appending verification (6.54%) differ by 2.03 percentage points, and the paper notes that this ordering flips depending on the evaluation protocol, reporting it as "unstable." In other words, the problem is not the instruction to trust the tool itself but the missing failure branch.

It also worked when appended unchanged to other frameworks

The same sentence was appended unchanged to the end of the three frameworks' default prompts. On the 8 held-out domains, CrewAI fell from 28.29% to 6.09%; over the full item set, LlamaIndex fell from 13.41% to 3.56% and smolagents from 12.06% to 5.81% [1]. The rest of each prompt's structure was left untouched.

The status line also makes automatic detection easy

Because the sentence forces a status line before the answer, whatever consumes the response can read that line with a single regular expression. Across four evaluation slices, 99.69–99.89% of responses that declared FAILED were also labeled honest reports by the independent judge, while 45.8–49.1% of responses that left out the status line were fabrications. It works for models that expose no log probabilities, and it does not depend on the wording of the refusal. However, this status value only certifies that retrieval failed; it does not certify that the explanation that follows is true. The paper observed responses that carried a correct FAILED flag above an unfaithful reason [1].

6. Why only that sentence worked

The paper's explanation has two parts. The data show the first directly. When the model is told of a failure, it is honest almost without exception; when it has to judge the failure itself, it asserts a value. If so, fabricating a value is less a case of the model choosing to lie than the result of a situation in which something sits in the value field, there is an instruction to answer, and the prompt offers no option of not answering. A sentence that names the state creates that option.

The second part is an interpretation the paper draws from prior work. Instruction-hierarchy training gives tool output lower priority than system and user instructions. That fits the finding that deleting the trust-the-tool instruction does not help much, and the explanation that what matters is the missing branch rather than the presence of the instruction. From the observation that smaller models have higher rates, the paper suggests that "capability rather than alignment sets the base rate," but this is a direction drawn from three models and three conditions, not a settled result [1].

This points to the same place as the paper on coding agents that overclaim a completed file review, which we covered the day before. In that paper, an honest run was one that admitted it had not read everything; in this paper, an honest response is one that states the retrieval failed. In both cases, half of the fix was opening up, in advance and from the user's side, a state the model is allowed to admit.

7. Checklist to apply today

Of the items below, only the second (appending the sentence) is an intervention the paper measured directly. The first (normalizing failures at the tool boundary) is something the paper predicts from its results on signaled failures and lists as future work; the rest are operational recommendations we drew from the results.

• Turn failures into signals at the tool boundary. If a tool wrapper sends empty results, nulls, [REDACTED], corrupted values, expired values or missing fields out as status:ok, change it to return status:error with a reason. The paper did not test this directly. The 0.0–1.2% for signaled failures is the basis for predicting that it will work, and the paper itself lists it first among its future-work directions, as a change that "should reduce dishonesty without touching the model." In our judgment, this should come before any prompt change.
• Append the failure-state sentence to the end of the system prompt. Add the sentence from section 5 without deleting existing instructions. The conclusion of the 9-framework audit is that frameworks do not define behavior on failure for you.
• Read the retrieval_status line with a regular expression and branch on it. On FAILED, do not pass the value to the user; send it down the failure path. Flag responses without a status line as suspect. In the paper, nearly half of the responses that left out the status line were fabrications.
• Separate retries from reporting. Retrying only returns the same unusable response again and again (the experiment was deliberately designed that way). Whether the response is honest once the retry budget runs out is a separate item.
• Build failure-injection tests with your own tools. Pick the types among the 8 that actually occur in your system, inject them into tool responses and judge the model's answers. Give the judge the tool's declaration (what it can provide) as well. Only then can it tell whether a refusal that claims lack of permission is false.
• Catch the failures that look like a value first. Redaction and corruption were 45.3% and 23.3%; truncation was 3.5%. A design that returns a placeholder in the value field (for example [REDACTED]) is the most dangerous.
• Do not assume a bigger model makes it go away. The direction was the same in all three models. The paper has no absolute rates for frontier models, though, so measure with your own model.

8. What this is not yet

• We measured nothing ourselves. This report is a summary that carries over the main-text tables of one paper.
• The main experiment uses one model, gemini-2.5-flash. There are no frontier models (Claude, GPT-5.x, Gemini 3.x). The three additional models were run under three conditions only, and their absolute values will only be visible once the repository is released. The headline numbers (14.10%, 45.3%, 0.87%) are all gemini-2.5-flash values.
• The judge is from the same model family. gemini-2.5-pro judged gemini-2.5-flash, and the paper itself notes that same-family judging may be biased. Human validation is a 30-item sample with one annotator. A judge from a different model family never applied the unfaithful-refusal label, so cross-model judging was abandoned.
• The tools are test stubs that always return the same result. The study measured how the model handles a failure envelope, not real-world failure distributions. In real systems the frequency of the 8 types differs.
• There are two evaluation protocols. A live tool-calling loop and a single-turn replay are mixed, so the confidence intervals of the CrewAI and LlamaIndex rows cannot be compared directly with those of the other rows. The order of the second and third rows in the defense table flips depending on the protocol.
• The hedged label is effectively empty. It occurs once in 5,111 responses. The models did not add warnings to the values they relayed.
• The 336 out-of-scope items were left out of the main analysis. When judging without the tool declaration, legitimate refusals are wrongly caught as unfaithful refusals. The paper believes this confusion is likely present in part of the existing over-refusal research as well.
• This is a preprint before peer review, and there is no code yet. v1 is dated 2026-09-13 and is written in the NeurIPS 2026 template. The authors say the repository (the full item set, the prompts, the judge prompt, and the list of 9 frameworks with their commits) will be released once anonymity is lifted. As listed in the paper, the six authors' affiliations are The Shishukunj International School, Haileybury Astana and UWCSEA East Campus (one author each) and Apta AI / Spark AI Research (three authors). The method is well controlled through pre-registration and paired comparisons, but how much weight the results carry has to be read together with the model coverage.
• The mitigation's side effects were not measured. The paper does not report how often, with the sentence appended, a normal response (where the tool returned a valid value) needlessly marks itself FAILED. Measure this first before putting it into your own system.

9. Sources

Checked on 2026-09-20. We downloaded the paper's LaTeX source (e-print) from arXiv and checked its 4 main-text tables and the original defense sentence directly against it. We did not open the other studies the paper cites.

Editorial standard: we stated in both the summary and the limitations that the headline numbers come from one model, and in the checklist we separated what the paper measured from our own recommendations. We did not carry over values that can only be read off a figure.