Designing Agents That Don't Fail — Part 1: Start With the Workflow

Traditional systems analysis defines what a system must do. Agentic systems analysis must also define what the system may decide, what evidence it needs, where its discretion ends, and how the organization will know whether it used that discretion well.

Why workflow comes first

In 2025, MIT researchers examined hundreds of enterprise generative-AI pilots and landed on a number that made headlines: roughly 95% delivered no measurable return [1]. Most of these projects didn't fail because the models weren't good enough. They failed because nobody designed the system around the model: the workflow the agent was supposed to live inside.

That failure mode is older than AI. I've seen plenty of conventional software projects go the same way. This is a systems analysis failure and it has a systems analysis fix.

When an organization decides to build an agent, the first conversation I hear almost always turns to model selection. The model is the most visible part of the system, and capability, cost, and latency all matter. But the MIT findings suggest that model choice is rarely the decision that separates the agents that work from the agents that don't. The better first question is: what work should this system perform, from trigger to outcome?

Drop an impressive model into a poorly understood process and you get an impressive demo, but not necessarily a dependable system. Production work needs state, permissions, validation, retries, recovery, escalation, and a durable record of what happened. None of that shows up just because a prompt describes it.

An agent is not the workflow

Consider a task I've met in several forms: onboarding a new customer. "Onboard the customer" sounds like one task, but it's really a sequence of activities:

• Collect required documents and information.
• Check whether anything is missing, inconsistent, or out of policy.
• Enter validated data into the appropriate systems.
• Draft customer communications.
• Route unusual or consequential cases to the right person.
• Record what was completed and what's still outstanding.

Some of these steps require interpretation, and others are deterministic. Some change external systems. Some can be automated safely, while others need approval. If you don't include these distinctions in your design, the agent has to invent the operating procedure while carrying it out. It will pass a carefully chosen demo and then fail when a document is missing, a tool is down, or a case falls outside the normal pattern. When researchers analyzed why LLM agent systems fail, they traced most failures back to gaps in specification and system design rather than to the models themselves [2].

Make the work visible

Start by documenting the business workflow, not the agent architecture. A useful workflow description answers questions like:

• What event starts the process? When is it complete?
• Which people, systems, data, documents, and policies participate?
• Which sources are authoritative for each kind of fact?
• Where are the rules stable, and where is judgment genuinely required?
• What exceptions occur in real cases, not just in the official process?
• What can go wrong, and how should the system recover or escalate?
• Which actions are reversible, consequential, regulated, or expensive?
• What evidence must be retained so the outcome can be reviewed?

In my experience, this analysis often reveals that you don't need a fully autonomous agent at all. The best design may be a conventional workflow with a few carefully bounded model-assisted steps. Models are strong precisely where procedural code struggles: extracting meaning from free-form text, classifying messy inputs, interpreting documents. Deployed as a point solution for one of those problems, a model can deliver most of the value without taking over the entire workflow. Anthropic's engineering team, who have seen a lot of these systems, give the same advice: the most successful implementations use simple, composable patterns, and complexity should be added only when it demonstrably improves outcomes [3]. In production, the simpler design is often the more capable system, because each component gets to concentrate on the work it does best.

In production, the simpler design is often the more capable system.

What changes, what doesn't

Agentic systems don't make established engineering practice obsolete. They still serve stakeholders, operate inside organizational boundaries, consume and produce data, integrate with other systems, encounter exceptions, and have to satisfy security, performance, cost, and ownership requirements. Traditional systems analysis remains the foundation.

What's different is that an agent can interpret ambiguous inputs, select tools, adapt its next step to intermediate results, and produce outputs that can't be specified exactly in advance. Anthropic's engineering guide distinguishes the two: workflows orchestrate models through predefined code paths, while agents direct their own process and tool use [3]. Traditional analysis primarily defines correct behavior. Agentic analysis must define a controlled range of acceptable behavior.

Decide where control flow should live

Workflow-first doesn't mean hard-code everything. It means understanding the work well enough to decide where control should live: in deterministic software, in a model-assisted decision, in an agent that selects the next step, or with a person. Control can move gradually from code to the agent; it doesn't have to be handed over all at once.

Classify every important step

Deterministic. Use ordinary software for validation, calculations, permissions, known business rules, state transitions, and anything else that can be specified reliably.

Model-assisted. Use a model for bounded interpretation, extraction, classification, comparison, summarization, or drafting when the inputs or outputs are genuinely ambiguous.

Agent-directed. Let the agent select an approved tool, decompose a task, request more evidence, or choose the next step when the correct path can't reasonably be known in advance.

Human-controlled. Keep approval, policy judgment, exception ownership, and consequential or poorly understood decisions with a person until you have evidence that a wider operating envelope is justified.

The classification drives everything downstream. Deterministic steps need conventional tests. Model-assisted steps need representative examples and quality criteria. Agent-directed steps also need limits on actions, time, cost, and repetition. Human-controlled steps need a review experience that surfaces the relevant evidence without forcing the reviewer to redo the whole task.

Use the simplest effective architecture

Autonomy is not the default destination. Start with the least complex design that can meet the outcome. A single model-assisted step may be enough. A fixed sequence works when the task decomposes cleanly. Routing works when inputs fall into recognizable categories. Parallel work helps when subtasks are independent. Iterative generation-and-review makes sense when the quality criteria are clear. Open-ended agentic control is justified only when the route and number of steps can't be predicted at all. This is the same escalation path Anthropic recommends in its catalog of agent design patterns [3].

Architectural complexity should be earned by measured improvement on real work.

Every increase in discretion adds potential value, but also latency, cost, failure modes, and evaluation burden. I've never regretted starting simpler than I thought I needed.

A practical analysis method

The sequence below combines traditional process analysis with the additional artifacts agentic behavior needs. Little of it will surprise anyone who has done systems analysis on conventional software; that familiarity is the point.

1. 01Define the outcome and system boundary. Replace broad ambitions like "use AI in customer service" with an observable result, such as resolving routine refund requests while routing policy exceptions to an employee. Identify the actors, systems, policies, risks, and ownership around that result.
2. 02Map the current workflow using real cases. Study ordinary work, awkward exceptions, incomplete inputs, and recovery behavior. The unofficial process often contains the most important requirements.
3. 03Classify each step. Mark work as deterministic, model-assisted, agent-directed, or human-controlled. Record why each step belongs in that category and what evidence would justify moving it later.
4. 04Choose where control flow lives. Use fixed orchestration when the path is known. Save runtime planning for the parts whose decomposition or next action can't reasonably be predetermined.
5. 05Turn capabilities into bounded tools. Expose structured, approved operations rather than broad system access. Define inputs, outputs, preconditions, permissions, limits, error behavior, and the evidence returned after an action.
6. 06Design context, state, and provenance. Specify what information is available at each decision, which sources are authoritative, what must survive retries or pauses, what may be remembered, and how claims stay linked to sources and timestamps.
7. 07Define the operating envelope. Set decision rights, resource budgets, stopping conditions, prohibited behavior, approval thresholds, and escalation triggers. Treat financial, computational, security, and operational authority as explicit requirements. This step gets skipped often enough that OWASP lists "excessive agency" among its top risks for LLM applications; the recommended mitigations are minimum necessary tools, minimum necessary permissions, and human approval for high-impact actions [4].
8. 08Design recovery and human control. Assume tools will fail, inputs will be incomplete, evidence will conflict, and some cases will be unfamiliar. Define how the workflow resumes, retries, asks for help, or hands ownership to a person.
9. 09Turn requirements into evaluations. Create realistic scenario suites, scoring rubrics, thresholds, and examples of unacceptable behavior before adding more autonomy. Evaluate the workflow as a whole, not just each model-assisted decision.

Where Part 2 picks up

This method gives you the skeleton: understand the work, classify every step, put control in the simplest component that can carry it, and bound whatever discretion remains. Steps 5 through 9 each involve substantial design work of their own, though. Tools have to translate ambiguous intent into safe operations, context and memory need explicit rules about what the agent sees and remembers, traces and evaluations have to catch failures that conventional telemetry misses, and human review has to amount to more than a checkbox. Part 2 will work through each of these in detail.

Part 2 is now available: Designing Agents That Don't Fail — Part 2: Design the Discretion.

References

1. [1]Jason Snyder, "MIT Finds 95% of GenAI Pilots Fail Because Companies Avoid Friction," Forbes, August 26, 2025. forbes.com
2. [2]Mert Cemri et al., "Why Do Multi-Agent LLM Systems Fail?" arXiv preprint arXiv:2503.13657 (NeurIPS 2025 Datasets & Benchmarks). arxiv.org/abs/2503.13657
3. [3]Anthropic, "Building Effective Agents," Anthropic Engineering, December 2024. anthropic.com
4. [4]OWASP GenAI Security Project, "LLM06:2025 Excessive Agency," OWASP Top 10 for LLM Applications, 2025. genai.owasp.org