Why Most Multi-Agent Systems Fail Even If the Evaluation Passes | Towards Data Science

I constantly encounter some version of this failure whenever agents are chained together in some form. Take, for example, a support ticket sorting system that is three nodes deep.

One classifies the incoming ticket, another retrieves the customer’s account history from the internal API, and the third drafts a resolution or escalation based on both.

Then he sets off. This works well in the demo version, which frankly, in my experience, doesn’t tell you much. This also works in the first few days of work; again, this tells you a little more, but still not enough.

At the same time, a complaint is received about a refund for canceling a subscription. The account history node calls the billing API, gets 200 back, and passes the payload downstream as if nothing happened.

Payload is empty. No bad data, no timeout, nothing that would show up as 500. Just an empty result set formatted exactly the same as a valid response because the account ID was mixed up two steps up and the billing service quietly returned nothing for an account it couldn’t match.

The drawing node never sees errors. He sees a well-formed JSON object with no records, decides that means “no payment history” and writes a perfectly polite email explaining that there is nothing to return.

It goes out, the wrong decision to return and that’s it. But so far no one has caught this, because nothing in this process has ever frozen. As for the system, it served its purpose.

I don’t think this particular scenario needs to happen to you for it to be worth your time.

If you’ve spent any time working with multi-agent systems in production, you’ve either already seen this version or will see it sooner or later.

All this is also not anecdotal. Datadog’s 2026 State of Artificial Intelligence Development report puts the production failure rate of AI queries at about 5 percent, and only about 60 percent of those are due to high-profile performance-related failures that you can actually see from the error code.

The rest is closer to what happened above: a request that completes but still fails.

Why doesn’t it catch anything?

Run this pipeline through a standard assessment package and it will pass successfully. The final result reads well, is grammatically clean and professionally worded.

There’s no reason for a person reviewing it for tone to note anything unless they accidentally cross-reference the actual account, which defeats the purpose of automating the review in the first place.

If you’re judging it based on resolution quality, it probably performs well too. Clear, polite and on topic.

It passes because each of these checks checks the same layer: the final text. None of them ask what happened between the second and third nodes.

The account history node did not fail loudly; it failed because it managed to return an incorrect result, and success is exactly what the eval function at the output level is rewarded for.

I think it’s worth sitting with this part longer than seems natural before you start fixing it.

Evaluating only the compiled output leaves you structurally blind to intermediate states that look correct but actually aren’t. This is not a gap that can be addressed with a better last node prompt. This is a blind spot that occurs where you decide to look first.

Evaluating the user interface, not the application underneath it

There’s an old comparison here that I keep coming back to. Nobody ships a compiled application and calls it tested because the login screen is displayed.

You test the layer underneath it, the request that supports login, the token it issues, and even the permission check it runs along the way.

The user interface is the last place where a bug shows up, not the first place you think to look for it.

Most production agent evaluation at this point is UI-only testing tied to a system that doesn’t even have a UI in the traditional sense.

The final text response is the only thing that is scored, mainly because it is the only thing that is easy to score.

You can review it by criteria, compare it line by line with a known correct answer, or have someone review it over a cup of coffee.

Tool calls, JSON passing between nodes, partial reasoning being passed forward: none of this is tracked unless something fails badly enough to leave a trace in the log somewhere.

And the costly failure mode was never loud. 500 – A server that openly admits that something is broken falls into a trap and escalates because the system already expects this kind of failure and has some plan for it.

The one that costs you is 200, a response that says everything is fine, attached to a payload that is structurally good but semantically garbage.

Architecture for Watching the Middle

So correction is not another rubric tacked on to the end. Some of the computation moves into the pipeline itself, right to the places where the output of one agent becomes the input of another agent.

I called it Interim State Assessment Architecturemainly because he needed a name and it stuck. The idea is for the lightweight grader to sit between agent nodes and not wait for the entire chain to complete.

In the case of ticket sorting that we talked about earlier, this is a checkpoint between the account history node and the composition node, and its only job is to figure out whether the transfer looks plausible.

Does the account ID in the payload actually match what was requested?

Does this look like a real search result or a default that the system silently reverts to when it can’t find what it’s looking for?

You don’t need a big model to make that judgment, and frankly, I’d argue against it.

A small local model serving as a watchdog between nodes, the same basic idea as using a model to evaluate the output of another model, is sufficient to identify problems at the shape level and likelihood level, and keeping it local prevents the added latency and cost from becoming a second version of the specific problem you’re trying to solve.

His verdict doesn’t actually have to be smart. It just has to be fast and binary: does this transfer look smart enough to move forward, or does it need to be flagged and stopped before the next node builds anything on top of it?

That’s all the work, nothing more.

Here’s what it actually looks like in code.

Start with the form of the transfer itself. Defining it as a real circuit using Pydantic instead of an arbitrary definition is what makes the watchdog work in the first place, since it gives the evaluator something specific to check for instead of guessing a new structure every time it’s called.

from pydantic import BaseModel, Fieldimport logginglogger = logging.getLogger("pipeline.handoff")# below this, the watchdog treats a "plausible" verdict as implausible anyway -# started at 0.5, got burned by a bunch of low-confidence-but-correct handoffs# in staging, moved it down until the false-halt rate stopped annoying peopleCONFIDENCE_FLOOR = 0.35class AccountHistoryPayload(BaseModel):    account_id: str    subscription_status: str    billing_records: list[dict] = Field(default_factory=list)    lookup_source: str  # which upstream system actually answered - saved us more than once when this was the only clueclass HandoffVerdict(BaseModel):    is_plausible: bool    reason: str    confidence: float

The watchdog itself is just a small feature, not some new service that you need to support and maintain.

It takes the outgoing payload, the request that created it, and asks the local model one narrow question instead of an open one.

Honestly, the thing about keeping the question narrow is that this is what allows it to remain cheap enough to actually stay on the critical path rather than becoming its own bottleneck.

import textwrap# local_grader: anything callable that takes a prompt and returns text.# we're running a distilled 1B model on the same box as the pipeline -# it doesn't need to be smart, it needs to be fast and not fall overGRADER_PROMPT = textwrap.dedent("""\    A downstream agent is about to receive this account history payload:    {payload}    It was requested for account_id: {account_id}    Answer strictly as JSON: {{"is_plausible": bool, "reason": str, "confidence": float}}    Flag it as implausible if the account_id doesn't match, if billing_records    is empty for a subscription marked active, or if the payload looks like    a default/fallback value rather than a real lookup result.""")def grade_handoff(request_account_id: str, payload: AccountHistoryPayload, local_grader) -> HandoffVerdict:    prompt = GRADER_PROMPT.format(payload=payload.model_dump_json(), account_id=request_account_id)    raw_response = local_grader(prompt)    try:        verdict = HandoffVerdict.model_validate_json(raw_response)    except ValueError:        # the grader itself can return garbage - if we can't parse its verdict,        # don't just shrug and let the handoff through, that defeats the point        logger.error("Grader returned unparseable output, blocking handoff: %r", raw_response[:200])        return HandoffVerdict(is_plausible=False, reason="grader output unparseable", confidence=0.0)    if not verdict.is_plausible or verdict.confidence < CONFIDENCE_FLOOR:        logger.warning(            "Handoff rejected for account_id=%s: %s (confidence=%.2f)\npayload=%s",            request_account_id, verdict.reason, verdict.confidence, payload.model_dump_json(),        )    return verdict

And yes, the orchestration is as unattractive as it gets; By the way, this was done on purpose. Somewhere in your pipeline there are already functions that process the classification and prepare the final decision.

The only new part here is the gate located between them, which adjusts the gears and boosts instead of sneaking the low-quality payload through to whatever the client-facing response is writing.

class HandoffRejectedError(Exception):    passdef run_pipeline(account_id: str, account_data: dict, local_grader) -> AccountHistoryPayload:    payload = AccountHistoryPayload(**account_data)    verdict = grade_handoff(account_id, payload, local_grader)    if not verdict.is_plausible:        # this replaces what used to be a wrong email going out quietly -        # a raised exception here is annoying, a refund that shouldn't exist isn't        raise HandoffRejectedError(f"account_id={account_id}: {verdict.reason}")    return payload  # safe to hand off to whatever drafts the resolution

There’s nothing unusual about this, and I’d be suspicious of anyone who dressed it up to make it sound like it is.

This is a pattern, one narrow evaluation criterion and an exception where previously there was a silent passage.

Value has never been in the sophistication of any particular item. It depends entirely on where you decide to place the check.

The practical benefit is that failures no longer happen silently. Instead of the bad email being sent three steps down from the actual problem, the pipeline stops right at the point of corruption, with the actual bad transmission being associated with any alert being triggered.

This turns a support escalation that costs you trust into a debugging session that costs you minutes.

What is it worth to you

None of this is free, and I’d rather be frank than sell you a strict upgrade with no downsides, because there’s not one, but three costs here, namely:

• Delay: Simple and clear. For a three-node pipeline, that’s two extra output calls right on the critical path, and they add up quickly if you’re building something where response time really matters.

• New fracture surface: The uncalibrated watchdog begins to reject perfectly good control transfers, trading silent corruption for another irritation, false stops that someone now has to check manually. Setting this threshold correctly requires actual iteration rather than a one-time set number.

• Solution: Deciding where a pass is truly worth evaluating and where you’re just adding overhead for its own sake depends entirely on how costly it is to get it wrong at that particular point.

Not every node needs a watchdog timer. The ones that are just before something external, sending an email or actually refunding, are the ones where the cost of detecting a bad handoff clearly outweighs the cost of the extra hop.

In other cases, you’re probably just increasing the delay for the sake of being more thorough.

···

Final Thoughts

If you’re using a multi-agent pipeline today and want to try it out without tearing it apart, don’t start at the beginning of the chain.

Trust me.

Start with one last internal communication before anything external happens, right before you send an email, record a note, and make a decision. This is the only boundary that should be set first.

Give it a week or two, see what the watchdog actually picks up, and let that tell you whether it’s worth putting it back into the pipeline.

You don’t need a full architecture on day one to get anything out of it.

The next time something breaks in your pipeline, ask yourself if your programs would have been able to detect it before the output itself looked wrong.

In most cases, to be honest, the answer is no. It is in this gap that the riskiest transmission in your system has been sitting all along, waiting for someone to look into it.

Your final result may be lying to you. The trajectory cannot.

···

Before you go!

I write about the engineering decisions that determine whether an artificial intelligence system can survive production operations. You can subscribe to my newsletter if you want more.

Contact me