Client profile
Chatwell, a UK customer service chatbot company with 70 staff.
Challenge and scope
The chatbot needs more useful answers while maintaining its safety criteria. The team needs consistent rankings at a scale its current reviewers cannot cover.
A fluent chatbot response can omit required information or sound persuasive while violating the client’s policy. Reviewers need a shared definition of helpfulness and a way to prioritize safety criteria when comparing responses. Inconsistent ranking rules can introduce noise into feedback across otherwise similar tasks.
Delivery workflow
Rank 18,000 response pairs or sets a month across four response categories using the client’s approved rubric. Sample reviewer agreement and route ranking disputes through quality review.
Present reviewers with the same user context and the approved scoring guidance, varying response order when the evaluation setup allows it. Record the preferred answer, rationale, and any safety flag. Route close calls and response pairs with competing quality and safety considerations to QA.
Create a small agreement set that spans all four response categories and the model’s languages in scope. Review frequent sources of disagreement and apply updated guidance to later batches. Keep a record of the approved rubric, dataset version, and decisions on disputed items.
Delivery team
Poland, about 14 FTE. 1 AI ops lead, 2 QA reviewers and 11 ranking specialists.
Systems and tools
Client RLHF platform, Label Studio, internal ranking console.
Client decision rights
The AI company controls model training, reward model design, and deployment.
Early milestone
At week 4, the ranking agreement reaches 83%.
Outcomes
Company-reported results, with the baseline, measurement window and supporting record described for each measure.
| Measure | Baseline | Result | Window | Supporting record |
|---|---|---|---|---|
| Helpfulness score | 3.2/5 | 3.8/5 | 5 months | Model evaluation report |
| Ranking agreement | 74% | 92% | 5 months | QA review sample |
| Safety violations | 2.1% | 0.4% | 5 months | Model evaluation report |
Interpreting the results
The helpfulness score improves from 3.2 to 3.8 out of five while ranking agreement rises and observed safety violations decline. Treat reviewer agreement as a measure of labeling consistency. Use the held-out evaluation to compare helpfulness and safety across the same categories and report the evaluation population.
Recommendations for new clients
For conversational AI teams, evaluate helpfulness and safety on a held-out set. Record the model, rubric, and data versions when interpreting model-level changes.
Provide common customer-service requests, examples of policy-sensitive responses, and the scoring instructions. Check reviewer agreement on the same pairs before scaling. Agree how ties, uncertainty, and safety flags are represented in the export, then have the model team check whether the output can be used as intended.