How we tested
- 120 intents across billing, shipping, returns, account, and policy questions.
- Tool-calling flows: knowledge retrieval + CRM lookup + status update.
- Metrics: deflection, accuracy, handle time, refusal correctness, cost per 1k tickets.
Headline results
Claude 4.6: Best policy adherence and complex deflection; slower than minis but safest.
GPT-5.x: Strong tool reliability, balanced quality/speed; great for workflows with actions.
Gemini: Competitive on straight Q&A; weaker on strict policy and tool precision in our tests.
GPT-4.1-mini: Lowest cost/latency; good for triage and simple intents.
Which model for which job
- Tier-1 triage + FAQs: GPT-4.1-mini or local small models for cost + speed.
- Policy-heavy with actions: Claude 4.6 or GPT-5.x with strict system prompts and approvals.
- Multi-turn with retrieval + tools: GPT-5.x excelled in tool calling stability; Claude strong on refusal safety.
Cost snapshot (per 1k tickets)
- GPT-4.1-mini: lowest, ideal for triage.
- GPT-5.x: mid, justified by tool reliability.
- Claude 4.6: higher, justified when policy risk is high.
Match the model to the workflow—don’t pay for reasoning where you just need speed, and don’t skimp on safety for policy-heavy actions.
FAQ
Did you include retrieval?
Yes—each model used RAG over a 300-article help center plus CRM actions.
What about hallucinations?
Claude had the fewest policy violations; GPT-5.x improved with refusal rules and approvals.
Can we mix models?
Yes—use minis for triage, premium models for complex actions, and route dynamically.
How often to retest?
Quarterly or on major model releases; keep regression suites and evals versioned.