A support copilot grounded in product telemetry, not the help centre.

01 — Client context

A mid-market SaaS company whose product surface had outgrown its help centre. Tier-1 agents were spending most of their time reading customer logs and replaying sessions before they could answer anything substantive.

02 — Operational problem

Off-the-shelf chatbots trained on the help centre were either wrong or generic — and customers had learned to bypass them. The real expertise lived in product telemetry, account configuration, and recent release notes, none of which a stock RAG setup could meaningfully reason over.

03 — System designed

An agent-assisted support workflow with two surfaces: a customer-facing copilot scoped to a curated set of question shapes, and an internal copilot that drafted responses for tier-1 agents using live telemetry, the customer's configuration, and recent incidents.

04 — Implementation shape

• Telemetry retrieval built first, against a frozen evaluation set of 600 historical tickets
• Customer surface launched only for question shapes where the eval pass rate exceeded 95%
• Agent-side copilot rolled out with thumbs-up / thumbs-down on every draft, feeding nightly eval re-runs
• Quarterly retirement review for any question shape whose pass rate drifted

05 — Guardrails, evals & governance

• Strict allowlist of customer-surfaced question shapes; everything else is routed to a human
• Telemetry redaction layer prior to any model call, audited monthly
• Hallucination eval pinned to release blockers, not dashboards

06 — Measurable result

• −42% blended tier-1 ticket cost over two quarters
• Customer satisfaction on AI-handled tickets one point above the human baseline
• Average tier-1 agent handle time down 31% on retained tickets