7 mistakes of production AI agents and human↔agent handoff
Most AI agents “break” not at the demo but in production: because of vague human↔agent handoff, a missing stop-list, blind KPIs, and knowledge without an owner — not because of a “weak model.”
Below are seven mistakes I most often review after go-live, and how to address them. The focus is on support, lead qualification, and internal ops agents. The numbers are order-of-magnitude figures from 2025–2026 pilots, not a promise that “your results will be the same.”
Mistake 1. An agent without a clear boundary of responsibility
Symptom: the bot answers “everything,” then confidently lies in an edge case.
| Bad | Good |
|---|---|
| ”Assistant for any questions” | Scope = top-N topics + explicit out-of-scope |
| The model decides on its own when to call a human | Escalation rules + stop-list |
| No list of forbidden actions | Money, legal promises, contract changes — human only |
Rule: the agent is a process executor, not a “universal employee.” The wider the scope without data, the higher the hallucination rate.
Mistake 2. Handoff = “pass it to chat” without context
If the operator receives an empty “the client wrote,” the handoff is harmful: the human wastes time digging around, and the client repeats their story.
Minimum escalation package
- Full dialogue history (not just the last message)
- Identified intent and why the escalation happened
- What the agent already promised / what data it collected
- Customer / order / ticket ID in the CRM
- Recommended next step for the human (not necessarily correct, but structured)
- Priority (VIP, money, complaints, SLA deadline)
| Handoff metric | Target after tuning |
|---|---|
| % of misrouted escalations | <12–15% |
| Time to first human response after escalation | per SLA, not “whenever someone noticed" |
| "Tell me again” repeats | → 0 ideally |
Mistake 3. KPIs only track ”% resolved by the bot”
Auto-resolution without quality is a trap. The agent can “close” dialogues with brush-offs.
| KPI | Why |
|---|---|
| % auto-resolution in scope | capacity |
| CSAT / rating on bot-closed dialogues | quality |
| Reopen rate within 48–72 h | false resolutions |
| Hallucination / factual error rate | trust |
| Cost per resolved dialog | economics |
| % of escalations with full context | handoff maturity |
In pilots, I plan for 40–60% auto-resolution in a narrow scope with CSAT ≥ baseline and reopen no worse than manual mode. If auto-resolution hits 80% but reopen spikes — that’s not a success.
Mistake 4. Knowledge “uploaded as PDF” and forgotten
Production dies from outdated cards: new tariff, old answer.
Knowledge ownership
| Role | Responsibility |
|---|---|
| Process owner | what is true, what can be promised |
| Knowledge base editor | cards, versions, dates |
| Operator / team lead | ”answer is outdated” signal from shifts |
| Developer | not the only one who “edits the prompt” |
Checklist:
- Each card has a source and a revision date
- Alert channel: “wrong answer” → ticket to the knowledge base within 24–48 h
- Freeze of critical wording (offer, refund, warranty)
- Rollback to the previous version of the knowledge base
Mistake 5. No shadow mode and no logs — only “we’ll see in production”
Without shadow mode and logs, you can’t tell a scenario bug from a model bug.
Minimum observability:
- Input/output log + tool calls (with personal data
Подпишитесь на @raisovich_news
Первыми получайте новые статьи об AI-автоматизации, нейросетях для бизнеса и создании сайтов. Без спама — только полезный контент.