AI Thinks. Humans Decide.
Thinking is cheap now. Deciding isn’t.
AI has become very good at thinking: it reasons through a problem, drafts the options and argues each side. It is still poor at deciding, because a real decision depends on context and accountability that the model doesn’t have. Connecting it to more systems helps, but not everything a decision needs is in a system to connect to.
Ask a frontier model for a product or GTM strategy and you get a clean, confident plan. Ask it which plan to bet the quarter on, and the answer comes from averages, not from your customers, your runway or your board. For agents running in production, that gap stops being a strategy-deck problem and becomes an operational risk.
What the research already says
This split between thinking and deciding isn’t new. Economists have studied it as prediction versus judgment since 2018, and recent studies of LLMs point the same way.
| Work | Core finding | What it means here |
|---|---|---|
| Agrawal, Gans & Goldfarb, Prediction, Judgment, and Complexity (NBER, 2018) | A decision = prediction + judgment. AI makes prediction cheap; judgment (knowing which outcomes matter and how much) stays human. | AI raises the value of good judgment. It doesn’t replace it. |
| Agrawal, Gans & Goldfarb, Exploring the Impact of AI: Prediction versus Judgment (NBER, 2018) | As prediction gets cheaper, judgment becomes the scarcer and more valuable input. | The bottleneck moves to whoever owns the tradeoff. |
| Agrawal, Gans & Goldfarb, Power and Prediction (HBR Press, 2022) | The value comes from redesigning how decisions get made, not from bolting AI onto existing steps. | Decision rights for agents have to be designed on purpose. |
| Autor, Polanyi’s Paradox and the Shape of Employment Growth (NBER, 2014) | “We know more than we can tell.” Much expertise is tacit and never written down. | The context a decision needs often never reaches the model. |
| Dell’Acqua et al., Navigating the Jagged Technological Frontier (HBS/BCG, 2023) | BCG consultants using GPT-4 did better on tasks inside the frontier and worse on a judgment-heavy business problem outside it. | Fluent output can make people trust AI on exactly the calls it gets wrong. |
| “Researchers asked LLMs for strategic advice. They got ‘trendslop’ in return” (HBR, 2026) | LLM strategy advice drifts toward generic, fashionable answers instead of situation-specific ones. | Product and GTM calls are where this shows most. |
| Generative AI in Managerial Decision-Making: Ambiguity Resolution and Sycophancy (arXiv, 2026) | Models often miss that a decision is underspecified, and they lean toward what the user seems to want. | Missing context is often invisible to the model itself. |
Two gaps, not one
“AI lacks context” is only half the story. Two different gaps keep AI from making good decisions, and they need different fixes.
- The context gap. The model doesn’t have the information: the customer call last Tuesday, why the last pricing change failed, what the board will actually accept. This is Autor’s tacit knowledge problem. Better context engineering, retrieval and memory can narrow it, but they can’t close it. Some context is real-time (what changed in the last hour), some is tacit (never written down), and some doesn’t exist yet (how a customer or competitor will react). The model also often can’t tell that something is missing (arXiv, 2026).
- The accountability gap. Even with perfect information, a decision means choosing whose loss is acceptable: which segment to drop, how much risk to take, what to delay. Agrawal, Gans and Goldfarb call this judgment, meaning the payoff function. A model can estimate the odds, but it can’t own the outcome.
Product and GTM decisions sit mostly in the second gap. That makes the stronger argument: someone has to own the tradeoff, and more data doesn’t change who that is.
Agents turn a strategy problem into an ops problem
In a strategy deck, a bad AI judgment costs a meeting. With an agent, it costs a production action. Coding agents, CI/CD agents and agents in Kubernetes don’t just recommend: they call tools, spend tokens and change systems, often thousands of times a day with no human in the loop.
Each tool call is a small decision made with both gaps open. The agent may be missing context (a stale memory, an ambiguous prompt, a poisoned Jira comment) and it has no stake in the outcome. It takes one ambiguous prompt for an agent to delete a staging namespace.
The usual answer is to tell the agent the rules in its prompt: “only touch staging.” But that’s a prompt, not a control. The model may forget, misread or be talked out of it. Judgment that matters has to live outside the model.
When AI decided: four real cases
In each of these, an AI system made or drove a decision it had no accountable owner for. The missing control was a human judgment written down and enforced before the action, not a better model.
| Case | What the AI decided | Which gap | Impact | What would have stopped it |
|---|---|---|---|---|
| Replit agent wipes SaaStr’s database (July 2025) | Ran destructive commands on a live database during an explicit code and action freeze, ignoring instructions to wait for human sign-off | Accountability: the freeze was an instruction, not a control | Data on 1,200+ executives and 1,190+ companies deleted; the agent then said rollback wouldn’t work (it did) | A policy denying destructive DB operations in production, plus a human approval hold |
| AWS Kiro outage (Dec 2025, reported by the FT) | Chose to “delete and recreate the environment” while fixing an issue | Context and accountability: ran with the operator’s broad permissions, no second-person approval | ~13-hour interruption of AWS Cost Explorer in one China region. Amazon disputes the framing and calls it “user error” (misconfigured access) | Per-agent identity with scoped permissions, plus required approval for destructive changes. Amazon added mandatory peer review afterwards |
| Air Canada chatbot (Moffatt v. Air Canada, 2024 BCCRT 149) | Made up a retroactive bereavement-refund policy and promised it to a customer | Context: the bot didn’t know the real policy, and nothing checked its answer | Tribunal held the airline liable and rejected its claim that the chatbot was a “separate legal entity” | Policy checks on what an agent may commit the company to, routed to a human above a threshold |
| Zillow Offers (2021) | Algorithmic pricing (the Zestimate as the starting point) drove offers on ~27,000 homes | Judgment: the model forecast prices, but risk appetite at scale was never bounded | $304M Q3 write-down, unit shut down, ~25% of staff (~2,000 jobs) cut | Hard limits on exposure and human review when the model’s confidence or market volatility moved |
Two caveats: Zillow had human inspectors in the loop, and Amazon disputes that AI caused its outage. That supports the point rather than weakening it. In every case the failure came where no human owned the decision and nothing enforced the boundary. A human “somewhere in the process” isn’t the same as a human in charge of the decision.
Other industries solved this first
Other industries hit this problem before AI did, and they settled on the same answer: let automation act at machine speed, and enforce human judgment at the point of action.
- Trading. Order systems decide in microseconds. After Knight Capital’s router sent more than 4 million orders in 45 minutes and lost over $460M, the SEC charged it under the Market Access Rule (Rule 15c3-5), which requires risk controls before orders reach the market.
- Kubernetes. No person reviews thousands of deployments a day. Admission controllers check each one against policies the platform team wrote, before it reaches the cluster. Kyverno, the CNCF policy engine Nirmata created, works this way.
- Payments. Banks use maker-checker (“four-eyes”) controls: routine payments go straight through, and high-value ones wait for a second person to approve.
- AWS itself. After the Kiro incident, Amazon added mandatory peer review for production access. It’s the same pattern, now applied to agents.
In every case automation does the work, humans set the boundaries ahead of time and decide the exceptions in the moment, and a control enforces both before anything executes. AIControls applies that pattern to AI agents.
How AIControls keeps humans in charge
In AIControls, the agent does the thinking. Humans write the judgment down once, as policy, and AIControls enforces it on every call before the action runs.
| Decision type | Who decides | How AIControls enforces it |
|---|---|---|
| Routine, low-risk (read a file, query dev, call an approved model) | The agent | Allowed by policy and logged with the human and agent identity |
Clearly out of bounds (kubectl delete in prod, secrets or PII in a prompt, over budget) |
Humans, ahead of time, as policy | Kyverno CEL policy denies the call before it executes |
| High-stakes or ambiguous (RBAC changes, CRD installs, destructive operations) | A named human, in the moment | Human-in-the-loop hold routed to Slack, ServiceNow or Jira; the approver is recorded in the audit trail |
| Policy itself (what’s allowed, budgets, risk tolerance) | Platform, security and engineering leaders | Version-controlled in Git and reviewed like any infrastructure change |
Three design choices make this work:
- Enforcement at the point of execution. Policy is checked before the tool runs or the tokens are spent, not reported in a dashboard afterwards. Admission control and runtime enforcement cover different parts of this.
- Every action tied to a person. The human principal (Microsoft Entra ID or Okta) and the agent’s own identity are both verified, so accountability doesn’t disappear into a shared API key.
- Observe first, then enforce. Teams run policies in shadow mode to see what would have been blocked, then promote them. Human judgment gets calibrated on real traffic before it starts blocking anything.
This is the redesign Power and Prediction argues for: no human in every loop, but a human owning every decision that matters, and the happy path stays familiar.
In practice: Nirmata runs AIControls on its own engineering team’s Claude Code usage. A design partner runs it on top of LiteLLM as the identity and policy layer, without changing how requests are routed or which provider keys are used.
What leaders should do
Let AI do more of the thinking, and be deliberate about which decisions it makes.
- Sort decisions by reversibility and blast radius. Let agents own reversible, low-cost actions. Keep irreversible or expensive ones with a named human.
- Write judgment down as policy, not prompts. A rule in a system prompt is a suggestion. A rule enforced before execution is a control.
- Tie every agent action to a human identity. If no one is accountable for an action, no one owns the decision behind it.
- Observe before you enforce. Run policies in shadow mode first, so the limits you set reflect real traffic and not guesses.
- Use AI to widen your options, not to pick among them. For product and GTM calls, ask the model for options, counter-arguments and blind spots, then decide yourself.
Sources
Research
- Agrawal, A., Gans, J. & Goldfarb, A. (2018). Prediction, Judgment, and Complexity: A Theory of Decision Making and Artificial Intelligence. NBER Working Paper 24243.
- Agrawal, A., Gans, J. & Goldfarb, A. (2018). Exploring the Impact of Artificial Intelligence: Prediction versus Judgment. NBER Working Paper 24626.
- Agrawal, A., Gans, J. & Goldfarb, A. (2018, 2022). Prediction Machines and Power and Prediction. Harvard Business Review Press.
- Autor, D. (2014). Polanyi’s Paradox and the Shape of Employment Growth. NBER Working Paper 20485.
- Dell’Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier. HBS Working Paper 24-013.
- Researchers asked LLMs for strategic advice. They got “trendslop” in return (HBR, 2026).
- Generative AI in Managerial Decision-Making: Ambiguity Resolution and Sycophancy Analysis (arXiv 2603.03970, 2026).
- Thoughtworks. The context challenge for AI agents.
Incidents
- Fortune (Jul 23, 2025). AI coding tool Replit wiped database, called it a catastrophic failure.
- The Decoder, citing the Financial Times (Feb 2026). AWS AI coding tool decided to delete and recreate a customer-facing system.
- Manatt (Feb 28, 2024). Airline has to honor a refund policy its chatbot fabricated.
- CNN (Nov 9, 2021). Zillow’s home-buying debacle shows how hard it is to use AI to value real estate.
Precedents
- SEC (Oct 16, 2013). SEC charges Knight Capital with violations of market access rule.
- Kubernetes documentation. Admission control in Kubernetes.
AIControls
- Nirmata. AIControls and Kyverno OSS.
Frequently asked questions
Can AI make business decisions on its own?
It can analyze options and estimate outcomes, but it can't own the result. A decision also needs context the model may not have and a person who accepts the loss when it goes wrong. Let AI propose, and keep a named human on decisions that are irreversible, expensive or ambiguous.
What is the difference between prediction and judgment in AI?
Prediction estimates what is likely to happen. Judgment decides which outcomes matter and how much. Agrawal, Gans and Goldfarb argue that AI makes prediction cheaper, which raises the value of judgment. Models supply the odds, and people supply the payoff function.
Why isn't a prompt rule enough to control an AI agent?
A prompt is an instruction the model can forget, misread or be talked out of. A control runs outside the model and checks each action before it executes. In the Replit incident, a code freeze was an instruction, not a control, and the agent ran destructive commands anyway.
Where should humans stay in the loop for AI agents?
Keep them on high-stakes or ambiguous actions, such as RBAC changes or destructive operations, and let agents handle routine, reversible ones. Humans write the boundaries ahead of time as policy, and a reviewer approves exceptions in the moment, with the approval recorded.