LangGraph Only for the Handoff: A Support Chatbot Where a Decision Model Picks Who Answers

Almost every multi-agent tutorial starts the same way: a generative LLM at the top with a prompt like “classify the user’s intent and answer in JSON.” It works. But every customer message turns into a full LLM call just to decide who answers. You get a label, not a confidence. And every now and then the JSON comes back broken.

I wanted to see what that architecture looks like with a different piece in the router’s seat: a decision model. It doesn’t generate text. You send it a state and a question with fixed options, and it returns one probability per option. That makes something simple possible that feels improvised with an LLM: when confidence is low, the bot asks before it acts.

I built a proof of concept of a post-sales chatbot for a fictional store, in Python, with LangChain and LangGraph. Total API spend: US$0.07. In this post I walk through the architecture, the code that matters, how I swapped providers by flipping a flag and, above all, where things broke.

The code is at github.com/gabrielcaiana/poc-langchain-roteamento-decisao.

Table of contents

• The conclusion first
• No rigid flows
• The architecture in one picture
• One turn, end to end
• What a decision model call looks like
• One contract, many providers
• The routing policy fits in four lines
• Subagents that actually hold a conversation
• What I saw when it ran
• The numbers, at the right size
• Where it broke
• What it cost and how I capped spending
• Limitations
• How to run it
• Next step

The conclusion first

• The router was the easy part. Across 76 evaluation turns, all three providers (MSD1, Jev, and an LLM classifier) got 100% of the non-ambiguous cases right.
• The value of a decision model isn’t “it’s more accurate.” It’s a fixed contract, lower cost and latency, and a probability you can actually threshold on. MSD1 cost US$0.0007 for the 76 turns versus US$0.022 for the LLM classifier, with a median of 232 ms versus 897 ms.
• 76 turns is a small sample. The gap between 98.7% and 96.1% is two turns. It can’t tell you which provider is better.
• The hard part is the subagent. The clarification agent sometimes answered instead of asking, and the billing agent opened a duplicate-charge ticket on an order where the data shows a single charge.

No rigid flows

I made one decision before writing the first line: the bot has no menus, no forms, no fixed sequence of steps. Customers write however they want.

Each subagent is a real agent (an LLM with tools) that decides on its own what to ask and when to act. On every message, the decision layer answers just one question: who handles this now?

That way, switching topics mid-conversation, sending a short follow-up (“and how long does it take?”), or saying something vague (“I’m not happy”) are handled by the design itself, without a special case for each.

There are five subagents (the names in the code are in Portuguese, since the store is Brazilian):

Subagent	Tools
returns (devolucao)	buscar_pedido, consultar_politica, abrir_solicitacao_devolucao
support (suporte)	buscar_pedido, atualizar_endereco
billing (financeiro)	buscar_pedido, consultar_politica, abrir_chamado_financeiro
clarification (clarificacao)	none
out of scope (fora_de_escopo)	none

Orders, store policy, and tools are all simulated. None of it is real customer data.

The architecture in one picture

The message goes through the decision layer, the threshold picks the subagent, and memory feeds the next turn.

LangGraph does only the handoff orchestration here. The graph has a classificar (classify) node and one node per subagent, and each subagent ends at END. The graph doesn’t model the steps of the support process. Where the conversation stands lives in the message history, and the subagent owns it.

So why LangGraph if it does so little? Because it gave me, for free, what a multi-turn chatbot needs and what I didn’t want to rewrite:

• typed state with reducers (add_messages to accumulate the conversation);
• a checkpointer per thread_id, which is the conversation memory;
• conditional edges for the handoff.

From LangChain I used the standard pieces: the classifier is a Runnable (it shows up in traces and accepts callbacks), tools are @tool, and subagents come from create_react_agent. The generative LLM is Gemini 3.5 Flash Lite, called through ChatOpenAI pointed at OpenRouter.

The graph state is small:

class Estado(TypedDict, total=False):
    messages: Annotated[list[BaseMessage], add_messages]   # conversation memory
    agente_atual: str | None                               # who handled the last turn
    decisao: dict                                          # current turn's decision
    trilha: Annotated[list[dict], operator.add]            # routing log for every turn

So is the wiring:

g = StateGraph(Estado)
g.add_node("classificar", classificar)
for nome in PROMPTS:
    g.add_node(nome, no_agente(nome))
    g.add_edge(nome, END)
g.add_edge(START, "classificar")
g.add_conditional_edges("classificar", lambda s: s["decisao"]["destino"], {n: n for n in PROMPTS})
return g.compile(checkpointer=checkpointer or MemorySaver())

One turn, end to end

A real turn: "by the way, I got charged twice via pix."

Taking a real message from the demo, in the middle of a conversation about a delivery:

1. The checkpointer loads the history and who’s currently handling it (suporte).
2. ClassificadorIntencao builds the state: the last 6 turns in compact form, the current specialist, and the customer’s last message.
3. The decision model returns financeiro=1.00.
4. The policy applies the 0.75 threshold and hands off to billing.
5. Billing calls buscar_pedido, consultar_politica, and abrir_chamado_financeiro.
6. The reply comes back as AIMessage(name="financeiro"), and the decision goes into the trilha (trail).

Sending the history to the decision model is what makes this design work. Without it, “yes, go ahead” or “is there a ticket number?” have no intent at all. In the evaluation set, the 12 follow-ups that depend on history were routed correctly by all three providers.

What a decision model call looks like

The main model is Microsoft-Decision-1 (MSD1), called through OpenRouter’s decisions endpoint, which is still in alpha. The question is of type choice, with a text criterion per option:

PERGUNTA = {"type": "choice",
            "instructions": ("Num atendimento de pós-venda, qual especialista deve responder à ÚLTIMA mensagem do cliente? "
                             "Use o histórico para entender mensagens curtas de continuação (ex.: 'e quanto tempo demora?')."),
            "criteria": INTENCOES}  # devolucao, suporte, financeiro, fora_de_escopo, each with a description

(The instruction asks which specialist should answer the customer’s last message, and tells the model to use the history for short follow-ups.)

The body of POST https://openrouter.ai/api/alpha/decisions is just {model, state, questions}. Here’s the state the classifier builds for a vague opener (“não tô satisfeito”, roughly “I’m not happy”):

{
  "especialista_atual": "nenhum",
  "historico": [],
  "ultima_mensagem_cliente": "não tô satisfeito"
}

And what came back, straight from the demo trace:

{
  "answers": {"intencao": {"probabilities": {
    "devolucao": 0.677255, "fora_de_escopo": 0.319913,
    "suporte": 0.001679, "financeiro": 0.001154}}},
  "usage": {"input_tokens": 201, "output_tokens": 1, "cost": 8.442e-06}
}

Confidence was 0.68, below the 0.75 threshold, so the turn went to clarification. That call cost less than a thousandth of a cent and took 268 ms.

One contract, many providers

I didn’t want the rest of the system tied to MSD1. Every provider implements a single method, decidir(estado, pergunta) -> Decisao. MSD1 and Jev (typesafe/jev-1.13, from TypeSafe) share the same endpoint, request body, and response shape, so they are the same class with a different model:

class ProvedorOpenRouterDecisao:
    def decidir(self, estado, pergunta):
        self.ledger.reservar(...)                                  # budget lock BEFORE the call
        estado = {k: v for k, v in estado.items() if k != "conversa_id"}
        r = self.transporte({"model": self.modelo, "state": estado, "questions": {"intencao": pergunta}})
        self.ledger.registrar(self.modelo, r["usage"]["cost"])
        probs = r["answers"]["intencao"]["probabilities"]
        ...

The transporte is injectable. That’s how the 13 tests run offline and how the demo simulates an MSD1 outage.

On top of that contract I built three things:

• LLM classifier (plan B). Gemini 3.5 Flash Lite with structured output via JSON schema, returning a 0 to 1 score per intent. I normalize it to sum to 1, but that isn’t a calibrated probability: it’s a number the LLM makes up.
• Fallback. ProvedorComFallback(primary, secondary) falls through to the next one on any error except a budget overrun, which has to stop everything. The msd1+fallback chain is msd1 → jev → llm.
• A/B. ProvedorAB picks the arm from a hash of the thread_id, so the whole conversation stays on one provider. Switching providers mid-conversation would muddy the comparison.

The choice lives in a flag:

"decisao": { "provedor": "msd1", "modelos": {"msd1": "microsoft/microsoft-decision-1", "jev": "typesafe/jev-1.13"}, "tau": 0.75 }

Valid values are msd1, jev, llm, msd1+fallback, and ab.

I tested the fallback for real: I simulated an HTTP 503 from MSD1, and all three turns of the conversation were handled by Jev, with the reason logged in the trace (fallback_motivo: RuntimeError: HTTP 503 ... (queda simulada)).

The routing policy fits in four lines

The whole handoff rule is a pure function, covered by tests:

def politica_roteamento(decisao, agente_atual, tau):
    if decisao.confianca >= tau:
        return decisao.intencao, "confiante"
    if agente_atual and decisao.intencao == agente_atual:
        return agente_atual, "continuidade"
    return "clarificacao", "baixa_confianca"

The second rule is continuity. If confidence is low but the most likely intent is the agent already handling the conversation, it stays with that agent. That avoids annoying clarification questions in the middle of a topic. And it doesn’t trap the conversation: if the customer changes topics with high confidence, the handoff happens on that same turn.

I picked the 0.75 threshold by hand; I didn’t calibrate it. And here’s an important detail: the same threshold doesn’t work across providers. MSD1 tends to go to 0.99 or higher. Jev spreads probability more on borderline cases (for example financeiro=0.62, devolucao=0.38). At τ = 0.75, MSD1 asked for clarification on 1 of the 76 turns and Jev on 5. The threshold is per provider, and switching providers in production would mean recalibrating it on a validation set.

Subagents that actually hold a conversation

Each subagent is a create_react_agent with its own prompt and tools. Two details took some work.

First: a subagent gets the dialogue (customer messages and final replies) but not its colleagues’ tool calls:

def _so_dialogo(msgs):
    return [m for m in msgs if isinstance(m, HumanMessage)
            or (isinstance(m, AIMessage) and m.content and not m.tool_calls)]

Without that filter, an agent inherited calls to tools it doesn’t even have, which caused unknown-tool errors.

Second: clarification receives the two most likely candidates in its prompt, so it can ask something specific instead of a generic “how can I help?”. The instruction is explicit: don’t answer the substance, ask one question, no menus. In the failures section you’ll see that wasn’t enough.

What I saw when it ran

Unedited recording of the interface (Streamlit + OpenRouter; the bot speaks Brazilian Portuguese). MSD1 probabilities for each turn on the right.

A topic switch mid-conversation, with MSD1 (customer messages translated in brackets):

👤 oi, meu tênis do pedido 10405 ainda não chegou, cadê?
   [hi, my sneakers from order 10405 haven't arrived yet, where are they?]
   ↳ [msd1] suporte=1.00 ... → suporte (confiante)
   ↳ ferramentas: buscar_pedido({"numero_pedido": "10405"})
👤 aproveitando, vi que me cobraram duas vezes esse pedido no pix
   [while I'm here, I saw I got charged twice for this order via pix]
   ↳ [msd1] financeiro=1.00 ... → financeiro (confiante)
   ↳ ferramentas: buscar_pedido(...); consultar_politica(...); abrir_chamado_financeiro({"valor": 349, "tipo": "estorno_duplicidade", ...})
👤 beleza, e em quanto tempo volta o dinheiro?
   [ok, and how long until I get the money back?]
   ↳ [msd1] financeiro=1.00 ... → financeiro (confiante)

And clarification doing what it should:

👤 não tô satisfeito                                  [I'm not happy]
   ↳ [msd1] devolucao=0.68 fora_de_escopo=0.32 ... | conf=0.68 τ=0.75 → clarificacao (baixa_confianca)
🤖 [clarificacao] Entendo. Para eu te ajudar da melhor forma, você gostaria de falar sobre a devolução de algum produto ou é sobre outro assunto?
   [I see. To help you best, would you like to talk about returning a product, or is it something else?]
👤 o fone do pedido 10418 parou de funcionar, quero trocar
   [the headphones from order 10418 stopped working, I want to exchange them]
   ↳ [msd1] devolucao=1.00 ... → devolucao (confiante)

There was also an “by the way, do you know if it’s going to rain tomorrow in São Paulo?” in the middle of an address change. It went to out of scope, and on the next turn (“ok lol. so the address is all set, right?”) it went back to support, which confirmed the address it had updated two turns earlier.

The numbers, at the right size

I wrote 76 fictional turns: openers, short follow-ups that depend on history, topic switches, informal language, and 8 cases I marked as ambiguous (for those, clarifying counts as correct). τ = 0.75.

Provider	Top-1 accuracy	Non-ambiguous	Clarification	Median latency	Cost (76 turns)
MSD1	98.7% (75/76)	100%	1.3% (1)	232 ms	US$0.0007
Jev	96.1% (73/76)	100%	6.6% (5)	195 ms	US$0.0017
LLM classifier (Gemini 3.5 Flash Lite)	98.7% (75/76)	100%	9.2% (7)	897 ms	US$0.022

Before drawing any conclusion, the sample size: 76 turns is not much. All three got every one of the 68 non-ambiguous cases right, and all the differences are in the 8 ambiguous ones, whose labels are debatable (I’m the one who decided what was ambiguous). The gap between 98.7% and 96.1% is two turns. With a set this size, that’s not evidence that MSD1 routes better than Jev.

MSD1’s only top-1 miss was, fittingly, an ambiguous one: “quero cancelar e receber o dinheiro” (“I want to cancel and get my money back”) went to support at 0.91, and I had labeled it billing. Both readings are defensible.

What the table shows more reliably is an order-of-magnitude difference in cost and latency:

• the LLM classifier cost more than 30× what MSD1 did and took almost 4× longer at the median;
• Jev was the fastest but came out about 2.5× more expensive than MSD1. The list price is the same (US$0.042 per million input tokens, free output), but Jev counts many more input tokens for the same request. In the demo traces, Jev calls ranged from 520 to 857 input tokens, and MSD1 calls from 201 to 364.

There’s also a difference in kind, not just in numbers: the LLM classifier’s “confidence” is a score it writes itself. It asked for clarification the most (7 times), and that doesn’t mean it was more uncertain. It means it wrote lower numbers.

To me, the evaluation shows that the architecture works with any of the three. It doesn’t show which one is the best router.

Where it broke

The failures that matter weren’t in the router.

Clarification doesn’t follow orders. Even with “do NOT answer the substance” in the prompt, Flash Lite sometimes answered instead of asking. In one run with Jev, the customer said “thanks! and how does the refund come back? I paid by card.” Confidence landed at 0.62 (billing versus returns) and the turn went to clarification, which answered right away: “Since you paid by card, the refund will be requested from the card issuer…”. With no tools, it answered without checking the policy. The route was fine. The agent just didn’t stick to its role.

What I’d do in the next version: take the LLM out of clarification and build the question from the two most likely candidates, or give it only the last message.

Billing opened a ticket without confirming it in the data. In the fictional database, order 10418 has one recorded charge. Still, when the customer said they’d been “charged again via pix,” billing called buscar_pedido, saw the order, and called abrir_chamado_financeiro with tipo: estorno_duplicidade (duplicate-charge refund). It happened in the interface recording (I left the GIF unedited precisely to show it) and again in the demos with MSD1, Jev, and A/B. It wasn’t a fluke: it’s how the agent behaves. On order 10405, which really does have two charges, the ticket was correct.

The right fix isn’t one more line in the prompt. It’s having the tool refuse the ticket when the data doesn’t confirm the duplicate charge. Business rules that matter belong in code, not in the prompt.

Smaller issues. Flash Lite once called atualizar_endereco with a placeholder address before the customer had given the new one (I fixed that with a rule in the tool’s docstring). In another turn, it repeated that the payment had been made via Pix after the customer said “card.” The system data was right, but the agent didn’t address the mismatch with what the customer said.

What it cost and how I capped spending

	US$
Total in the local ledger (479 calls, including the GIF recording)	0.0709
Gemini 3.5 Flash Lite (subagents + LLM classifier)	0.0667
Jev	0.0029
MSD1	0.0013

Almost all the spend went to the generative LLM. Routing with decision models cost a little over a tenth of a cent.

The POC had a US$3 cap, with a hard stop at US$2.80. The ledger reserves the maximum estimated cost before each call and records the actual usage.cost afterwards. If the reservation would exceed the limit, the call never goes out. And that budget error is the only one the fallback doesn’t swallow: if the money’s gone, falling through to the next provider would only spend more. In the end, the ledger matched the difference in the OpenRouter account balance before and after.

Limitations

• The data is 100% fictional. The 76 turns and their labels were written with LLM help and were not reviewed by a human. The ambiguous cases reflect my own judgment.
• The sample is small and has no statistical power to compare providers.
• The tools are simulated. There’s no customer authentication and no real side effects.
• Memory (MemorySaver) is in-process. In production I’d swap it for a persistent checkpointer such as Postgres or Redis.
• The decisions endpoint is in alpha. The contract may change, which is exactly why the fallback to an LLM classifier exists.
• There’s no handoff from inside a subagent. Billing can’t hand the conversation back to returns. Every switch goes through the classifier on the next turn.

How to run it

python -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
export OPENROUTER_API_KEY=...            # never goes into the repo
pytest -q                                 # 13 tests, offline
python scripts/demo.py --provedor msd1    # also: jev | llm | ab | msd1+fallback --simular-queda-msd1
python scripts/avaliar.py                 # routing evaluation msd1 × jev × llm
python scripts/chat.py --provedor jev     # terminal chat
streamlit run app.py                      # web interface

The repo has full traces for every turn (probabilities, tools, cost, and latency), terminal outputs, and the evaluation summary under evidencias/.

Next step

My takeaway: with a decision model, the router becomes the cheap, predictable part of the system, and the hard part is still the agent that takes action. If I were taking this to production, routing would be the least of my worries. The real work would be moving business decisions out of the prompt and into the tools.

If you already run a multi-agent bot with an LLM as the router, here’s a cheap test you can do today: take a log of a few hundred messages, run your router and a decision model side by side, and compare three things: agreement, latency, and cost. And look closely at where confidence drops below 0.75. That’s where your bot should be asking, not guessing.

The code is at github.com/gabrielcaiana/poc-langchain-roteamento-decisao.

Follow me on LinkedIn and GitHub for the next experiments.