Client profile
Lexica AI, a US enterprise legal AI company with 90 staff.
Challenge and scope
Reviewers disagree on 22% of cases in the baseline, and teams need a consistent way to score accuracy and safety across model versions.
Two reviewers may agree on whether an answer reads well and disagree on whether a legal claim is grounded in the reference. If the rubric leaves that distinction unclear, results become difficult to compare across versions. Reviewers also need a consistent way to flag unsupported citations and cases outside their assigned scope.
Delivery workflow
Run 4,200 evaluations a month against six rubric dimensions across three model versions. Calibrate reviewers on shared examples and send ambiguous cases to the assigned reviewer lead for adjudication.
Calibrate reviewers on a shared set of queries with known reference passages and annotated scoring examples. Score each dimension against the client’s rubric and attach evidence for a disputed or unsupported claim. Record the model version and rubric version before sending disagreements to the reviewer lead.
Keep adjudication decisions with the rationales and the references used by reviewers. Use them to explain recurring disagreements and update calibration materials after approval. Check agreement and evidence quality within individual dimensions, including examples that should trigger further review.
Delivery team
Poland, about 11 FTE. 1 eval lead, 2 QA reviewers and 8 calibrated reviewers.
Systems and tools
LangSmith, client eval platform, Google Sheets.
Client decision rights
The AI company owns rubric design, model selection, and deployment.
Early milestone
The week 4 record shows 86% agreement between reviewers.
Outcomes
Company-reported results, with the baseline, measurement window and supporting record described for each measure.
| Measure | Baseline | Result | Window | Supporting record |
|---|---|---|---|---|
| Inter-rater agreement | 78% | 94% | 4 months | Evaluation platform export |
| Evaluation throughput | 900 / month | 4,200 / month | 4 months | Evaluation platform export |
| Hallucination detection rate | 71% | 93% | 4 months | Evaluation platform export |
Interpreting the results
The increase from 78% to 94% inter-rater agreement is meaningful with the same agreement method and an appropriately difficult shared set. Measure hallucination detection against responses with known unsupported claims and report the sample size. Compare model versions on a common query set and stable rubric.
Recommendations for new clients
For teams commissioning human evaluation, test reproducibility on a shared labeled set. Report the agreement method, rubric version, and how reference hallucinations were identified.
Start with the legal use cases the product covers most often and a set of contentious edge cases. Provide the approved reference material, rubric, sample scores, and adjudication owner. Ask reviewers to score the same items independently and use the disagreements to improve the rubric and onboarding.