Designing Agents That Don't Fail — Part 2: Design the Discretion

The central addition is discretion. Systems analysis must decide how much control flow may be chosen at runtime — and build the evidence, boundaries, and feedback needed to trust those choices.

In Part 1, I argued that agents usually fail for systems reasons rather than model reasons, and laid out a workflow-first analysis method: define the outcome, map the real workflow, classify every step as deterministic, model-assisted, agent-directed, or human-controlled, and put control in the simplest component that can carry it reliably.

Several of those steps deserve a closer look. This part covers tool design, context and memory, quiet failures, evaluation, and where durable infrastructure and human judgment fit into the overall control system.

Tool design is systems analysis

A tool is more than an API made available to a model. Anthropic's engineers describe tools as "a contract between deterministic systems and non-deterministic agents" [1], which is a useful way to think about them. A technically correct interface can still be unusable by an agent if its purpose, parameters, boundaries, or failure modes are unclear.

For each tool, the analyst should define:

• A narrow purpose, and a name that communicates it.
• Structured inputs with validation, defaults, examples, and explicit units.
• Preconditions and permissions checked outside the model.
• Outputs that expose both the result and decision-relevant evidence.
• Clear error responses that distinguish retryable, correctable, and terminal failures.
• Idempotency, side-effect boundaries, and confirmation rules for consequential actions.
• Quotas, cost limits, concurrency controls, and approval thresholds where appropriate.

Most existing systems I've worked with needed wrappers, schemas, and more predictable error behavior before they were safe to hand to an agent. It's tempting to file this work under integration plumbing, but it's how an organization translates broad requests into constrained, testable operations.

Context and memory need an explicit design

Traditional analysis identifies records, fields, retention rules, and systems of record. Agentic systems add a less tidy category: the context available when a decision is made. Context may include the current request, retrieved documents, tool results, instructions, earlier decisions, working summaries, and conversation history.

What the agent sees strongly influences what it does, yet more context is not always better. Models have a limited attention budget, and performance degrades as context grows; researchers call this "context rot" [2]. Old, irrelevant, duplicated, or untrusted information can distort a decision. The analyst should specify:

• Which source is authoritative for each kind of fact.
• What information should be retrieved for each step, and how relevance is determined.
• How conflicting, stale, missing, or low-quality evidence is represented.
• Which working state must survive a retry, pause, handoff, or long-running task.
• What may be remembered across cases, users, or time, and for how long.
• How every important claim stays attached to its source, retrieval time, and transformation history.

Design for semantic failure

Traditional systems usually fail visibly: an exception is raised, a constraint is violated, a service returns an error, a calculation can be checked. Agentic systems can fail quietly. An agent can pick the wrong source, misread a document, overlook contradictory evidence, or recommend a permitted but inappropriate action. The output may be fluent and structurally valid, and every tool call may have succeeded.

Operationally the run looks healthy, but semantically it's wrong. The failures that matter most never throw an error.

Conventional telemetry therefore remains necessary but is not sufficient [3]. A production trace should retain the decision-relevant state needed to evaluate, reproduce, and challenge the result, including:

• The evidence available when an important decision was made.
• The tools, sources, and data versions used.
• Important classifications, structured judgments, and confidence indicators.
• Whether required checks and coverage criteria were completed.
• Why the workflow stopped, retried, escalated, or requested approval.
• Which deployed instructions, tools, and model configuration were active.

The trace doesn't need to preserve unlimited internal reasoning, only the facts, actions, intermediate artifacts, and decision summaries needed for audit and debugging. The final answer should be the last layer of a traceable process that leaves other artifacts behind it.

Requirements become evaluation criteria

A deterministic requirement can usually describe one expected behavior. Agentic requirements lean on words like accurate, complete, relevant, appropriate, or well-supported. Those words are not implementation specifications. They have to be translated into scenarios, scoring criteria, thresholds, and examples of unacceptable behavior. Current work on agent evaluation is built around the same shift: judging trajectories, tool choices, and outcomes rather than single outputs [4].

A useful evaluation set includes routine cases, ambiguous cases, rare exceptions, incomplete inputs, conflicting evidence, tool failures, permission boundaries, and adversarial requests. Reviewers can score several dimensions independently:

• Was the request and desired outcome understood correctly?
• Was sufficient and appropriate evidence gathered?
• Were claims and actions supported by that evidence?
• Were tools selected and used correctly?
• Did the workflow respect permissions, limits, and prohibited actions?
• Should the case have been escalated or stopped earlier?
• Was the explanation useful to the intended reader?
• Did the whole workflow achieve the business outcome at acceptable cost and latency?

And evaluation is continuous: deployed instructions, tools, data sources, models, and user behavior all change, so quality has to be sampled and reassessed in production.

Keep durable workflow responsibilities outside the agent

The production agents I've seen work best contain a surprisingly small amount of agentic behavior. The model might interpret a request, extract parameters, select among approved tools, decide whether more evidence is needed, and explain a result. Conventional infrastructure should still own trusted computation, persistent state, long-running execution, retries, scheduling, notifications, audit records, permissions, and resource accounting.

This separation matters most for work that spans minutes, hours, or days. A job should be able to pause, resume, retry, and report completion without keeping one model interaction alive. This is the gap that durable-execution platforms fill for long-running agentic work [5]. State transitions should be durable and inspectable. The agent should receive the state it needs for the next decision, rather than being expected to remember the entire history.

The same separation makes the system easier to change. Models, instructions, and tools can be replaced or compared at one decision point while the surrounding workflow, state, and evaluation cases stay stable. Operating logic lives in the system rather than in a single prompt or one person's memory.

Human involvement is part of the control system

A promise to keep a "human in the loop" is not a design. Decades of human-factors research on automation bias show that people who supervise mostly reliable automation tend to over-trust it and stop checking [6]. A person who receives a polished recommendation without supporting evidence may simply approve it. A reviewer who has to repeat all of the agent's work gains little from the automation. A poorly timed approval step adds delay without reducing risk.

Define the human role precisely:

• Supply missing information or resolve an ambiguous request.
• Choose among plausible interpretations.
• Approve an external, irreversible, regulated, or high-cost action.
• Handle cases beyond the agent's authority or competence.
• Review a sample of routine cases for quality and drift.
• Correct the result, and feed the correction into policy, tooling, or evaluation updates.

The review interface should show the evidence, uncertainty, proposed action, policy constraints, and relevant history. It should also record the disposition and what happens next. Treated this way, human review becomes part of the system's feedback and governance rather than a disclaimer attached to its architecture.

An evidence-heavy workflow example

These principles are easiest to see in evidence-heavy work: scientific research, financial analysis, compliance review, due diligence, or any task that gathers many sources and produces a consequential conclusion. A general workflow might:

1. 01Resolve the question and entities. Convert varied language into stable identifiers, scope, and explicit completion criteria.
2. 02Plan the evidence coverage. Select required source classes and optional branches based on the question.
3. 03Retrieve and normalize evidence. Use deterministic connectors and preserve failures, timestamps, and source identity.
4. 04Analyze bounded evidence units. Apply the appropriate criteria to each document, record, or evidence class.
5. 05Aggregate across results. Identify agreement, contradictions, gaps, and uncertainty without losing links to the underlying material.
6. 06Check completion and escalate. Determine whether coverage is sufficient, whether more work is justified, and whether expert review is required.
7. 07Produce a reviewable conclusion. Present the answer, evidence, limitations, unresolved questions, and provenance together.

The agent contributes interpretation, adaptive planning, comparison, and explanation. The surrounding system contributes identifiers, source access, normalization, state, completion tracking, evidence retention, and reproducibility. Neither side is sufficient alone.

The core analysis deliverables

Traditional artifacts (process maps, requirements, data models, interface specifications, security analysis, acceptance tests) remain useful. Agentic projects usually need several additions:

A final review checklist

• Can the workflow be described clearly from trigger to outcome without using "the agent decides" as a substitute for analysis?
• Is every important step assigned to the simplest component that can perform it reliably?
• Are runtime decisions bounded by evidence, permissions, cost, time, repetition, and stopping conditions?
• Do tools translate broad intent into structured, approved operations with inspectable results?
• Can the system distinguish tool success from task success?
• Can a reviewer reconstruct the evidence and state behind a consequential conclusion or action?
• Are human approvals placed where they reduce risk, with enough context to support a real decision?
• Do evaluations cover the whole workflow as well as individual model-assisted steps?
• Can the system pause, retry, resume, escalate, and change components without losing durable state?
• Is there evidence that additional autonomy improves the outcome enough to justify its added complexity?

Agentic analysis extends systems analysis

Traditional systems analysis tells us how work, data, people, rules, and software fit together to produce an outcome. Agentic systems analysis doesn't replace that discipline. It adds a new problem: a probabilistic component may interpret a situation and select behavior that was never fully enumerated in advance.

The analyst must therefore design not only the expected process, but the discretion inside it: the available evidence, bounded tools, decision rights, stopping conditions, evaluation criteria, recovery paths, human controls, and the durable trace around every important choice.

Organizations that can answer both questions can use agents for ambiguous work without giving up the discipline that production systems require.

References

1. [1]Anthropic, "Writing Effective Tools for Agents," Anthropic Engineering, September 2025. anthropic.com
2. [2]Anthropic, "Effective Context Engineering for AI Agents," Anthropic Engineering, September 2025. anthropic.com
3. [3]Fiddler AI, "Why Do AI Agents Fail Silently in Production?" Fiddler AI Blog. fiddler.ai
4. [4]Confident AI, "LLM Agent Evaluation: Tool Calling, Task Completion, Reasoning, and Trace-Based Evals," Confident AI Blog, 2026. confident-ai.com
5. [5]Temporal Technologies, "AI Applications & Agents with Temporal." temporal.io
6. [6]Raja Parasuraman and Dietrich H. Manzey, "Complacency and Bias in Human Use of Automation: An Attentional Integration," Human Factors 52, no. 3 (2010): 381–410. sagepub.com