When AI Sounds Most Useful Is When It Should Be Saying 'I Don't Know'
A new arXiv paper shows AI caution collapses when prompts push for a clear business answer — from 91% careful in researcher voice to under 10% in executive voice. What enterprise AI governance teams should change.
The 91% to 7% drop that should worry every AI governance committee
A new paper from arXiv (2606.24370) measured a pattern most enterprise AI deployments have already started to feel: when an LLM is asked the same causal question in two voices, its honesty drops by an order of magnitude.
Asked like a researcher, the model stayed careful 91 to 100 percent of the time. It named weak evidence and refused to overclaim. Asked like a business user who needed a recommendation — the exact phrasing that lands in strategy decks and Slack threads every day — caution collapsed to 7 to 18 percent. When the prompt pushed explicitly for a clean answer, only one response in 200 stayed cautious.
The model did not get dumber. It got more eager to sound useful. And that distinction is the entire problem for an enterprise rolling out AI in rooms where a confident wrong answer does real damage.
Where the risk actually lives in an enterprise
The paper's framing matters more than the percentages. The dangerous rooms are not the ones where AI is asked to write a draft or summarize a call. They are the rooms where someone needs to know what to do next: pricing, hiring, clinical decisions, regulatory posture, capital allocation, vendor selection.
In those rooms, the prompt almost always carries the pressure of a decision. The user has a deadline, a board meeting, or a customer waiting. They are not asking what the model thinks — they are asking what the model recommends. And the paper shows that the recommendation-mode prompt is precisely the one that strips out the model's calibrated uncertainty.
For an AI governance program, this means the failure mode is not the hallucination you can catch on review. It is the confident, fluent, on-topic answer that reads like the right call — except the underlying evidence was thin to begin with.
What changes inside an enterprise AI deployment because of this paper
Most AI governance frameworks were written assuming the model is the unreliable actor and the human is the safety net. The paper inverts that assumption for decision-mode prompts. The model is more reliable when nobody is pressing it, and less reliable precisely when a human most needs it to push back.
That changes three things inside an enterprise rollout.
1. Confidence calibration has to be a first-class output, not a footnote. If a model cannot say "I am 40 percent confident on this, here is what would change my mind" in decision mode, it should not be answering in decision mode. Procurement language should require this — and most RFPs still do not.
2. Prompt design is now a risk control. The same model behind the same API is materially safer when called by an analyst than when called by an executive in a hurry. That is a governance problem, not a UX preference. It belongs in the change-management plan, the training curriculum, and the access-control matrix.
3. The human-in-the-loop has to be moved earlier in the workflow. A reviewer who sees the AI's draft and edits the language is not catching the overclaim — they are polishing it. The check has to happen on the question itself: did the prompt deserve a recommendation, or did it deserve a structured uncertainty response?
A practical framework for adopting decision-mode AI inside an enterprise
Numbered rollout sequence. Four steps, each with a measurable outcome.
1. Audit the prompts already in production. Pull the last 90 days of AI interactions from the highest-stakes workflows — pricing, hiring, clinical, regulatory. Tally how many were framed as "tell me what to do" versus "what does the evidence say." That ratio is your current risk surface.
2. Pick one workflow with a known answer first. Validate the tool against an existing CI source. The point is to build trust in what you do not — not to confirm what you already know. Use it to measure the model's honest-disagreement rate before any business-critical rollout.
3. Require confidence language in every decision-mode response. Treat absence of a confidence statement as a defect. Build the prompt template to require "I am X percent confident because Y" or "I do not have enough evidence to recommend." This is a one-line change to the system prompt, but it changes the entire downstream workflow.
4. Move the human review upstream of the executive summary. The analyst or operator who edits the AI's output should be reviewing the question framing, not the prose. If the prompt was a decision-mode prompt and the model did not flag uncertainty, that response does not reach the executive without an explicit analyst override.
The governance questions that come up on day one
Every enterprise AI governance lead should be asking three questions this quarter.
Where does the decision-mode prompt live? In most enterprises, it lives wherever a senior leader has a Slack message, an email, or a Notion page open. That is not a workflow you can centrally govern — it is a behavioral pattern you can only train for and instrument against.
What is the marginal cost of an overconfident wrong answer? The honest answer is that it depends on the room. A pricing recommendation that is 10 percent off is recoverable. A clinical recommendation that is 10 percent off is not. The governance bar should be set by the room, not by the average use case.
How do you measure this in production? You need a metric for calibrated uncertainty in decision-mode prompts — not just for accuracy. A model that is 80 percent accurate and flags uncertainty 80 percent of the time is materially safer than a model that is 80 percent accurate and flags uncertainty 10 percent of the time. The second one is the dangerous one.
Where this lands inside an enterprise AI roadmap
Place this finding in the broader pattern of "tools that change the economics of a judgment function rather than replacing it." Pricing intelligence, talent signal monitoring, regulatory horizon scanning — each of these followed the same trajectory over the last 18 months. A procurement process that used to take a quarter now takes a credit-card decision.
The strategic reframe is this: the question is no longer whether AI can answer your executive's question. It clearly can. The question is whether the answer it gives in decision mode is calibrated enough to act on — and whether your organization can tell the difference in the three seconds between the response landing and the executive deciding.
For an enterprise AI strategy that is more than a slide deck, the work for 2026 is building the workflow that catches decision-mode overconfidence before it costs a board meeting, a patient outcome, or a regulatory finding. The model is not the problem. The prompt pressure is the problem. And the prompt pressure is the part your governance program can actually control.
Book a strategy consultation to map your decision-mode AI workflows against the calibration standards now viable in 2026.