Case study · AI data operations

Human preference ranking for chatbots

Company-reported case study

Actigy BPO ranked response preferences for Chatwell’s reinforcement learning from human feedback (RLHF) work across three languages. The five-month case compares a 3.2/5 baseline with a 3.8/5 reported helpfulness score.

Actigy BPO confirms the engagement and supporting records. Results apply to the stated scope and measurement windows.

Helpfulness score

Result

3.8/5

Baseline: 3.2/5

Window: 5 months

Supporting record: Model evaluation report

Ranking agreement

Result

92%

Baseline: 74%

Window: 5 months

Supporting record: QA review sample

Safety violations

Result

0.4%

Baseline: 2.1%

Window: 5 months

Supporting record: Model evaluation report

Client profile

Chatwell, a UK customer service chatbot company with 70 staff.

Challenge and scope

The chatbot needs more useful answers while maintaining its safety criteria. The team needs consistent rankings at a scale its current reviewers cannot cover.

A fluent chatbot response can omit required information or sound persuasive while violating the client’s policy. Reviewers need a shared definition of helpfulness and a way to prioritize safety criteria when comparing responses. Inconsistent ranking rules can introduce noise into feedback across otherwise similar tasks.

Delivery workflow

Rank 18,000 response pairs or sets a month across four response categories using the client’s approved rubric. Sample reviewer agreement and route ranking disputes through quality review.

Present reviewers with the same user context and the approved scoring guidance, varying response order when the evaluation setup allows it. Record the preferred answer, rationale, and any safety flag. Route close calls and response pairs with competing quality and safety considerations to QA.

Create a small agreement set that spans all four response categories and the model’s languages in scope. Review frequent sources of disagreement and apply updated guidance to later batches. Keep a record of the approved rubric, dataset version, and decisions on disputed items.

Delivery team

Poland, about 14 FTE. 1 AI ops lead, 2 QA reviewers and 11 ranking specialists.

Systems and tools

Client RLHF platform, Label Studio, internal ranking console.

Client decision rights

The AI company controls model training, reward model design, and deployment.

Early milestone

At week 4, the ranking agreement reaches 83%.

Outcomes

Company-reported results, with the baseline, measurement window and supporting record described for each measure.

Reported results: Human preference ranking for chatbots
MeasureBaselineResultWindowSupporting record
Helpfulness score3.2/53.8/55 monthsModel evaluation report
Ranking agreement74%92%5 monthsQA review sample
Safety violations2.1%0.4%5 monthsModel evaluation report

Interpreting the results

The helpfulness score improves from 3.2 to 3.8 out of five while ranking agreement rises and observed safety violations decline. Treat reviewer agreement as a measure of labeling consistency. Use the held-out evaluation to compare helpfulness and safety across the same categories and report the evaluation population.

Recommendations for new clients

For conversational AI teams, evaluate helpfulness and safety on a held-out set. Record the model, rubric, and data versions when interpreting model-level changes.

Provide common customer-service requests, examples of policy-sensitive responses, and the scoring instructions. Check reviewer agreement on the same pairs before scaling. Agree how ties, uncertainty, and safety flags are represented in the export, then have the model team check whether the output can be used as intended.

Scope this workflow for your operation

Use the documented workflow, client-held decisions and operational checks to outline a pilot for your own workload.