Case study · AI data operations

Human evaluation for legal language models

Company-reported case study

Actigy BPO calibrated human reviewers for Lexica AI’s enterprise legal product. Reported inter-rater agreement moved from 78% to 94% over four months, with rubric design and AI-model selection held by the client.

Actigy BPO confirms the engagement and supporting records. Results apply to the stated scope and measurement windows.

Inter-rater agreement

Result

94%

Baseline: 78%

Window: 4 months

Supporting record: Evaluation platform export

Evaluation throughput

Result

4,200 / month

Baseline: 900 / month

Window: 4 months

Supporting record: Evaluation platform export

Hallucination detection rate

Result

93%

Baseline: 71%

Window: 4 months

Supporting record: Evaluation platform export

Client profile

Lexica AI, a US enterprise legal AI company with 90 staff.

Challenge and scope

Reviewers disagree on 22% of cases in the baseline, and teams need a consistent way to score accuracy and safety across model versions.

Two reviewers may agree on whether an answer reads well and disagree on whether a legal claim is grounded in the reference. If the rubric leaves that distinction unclear, results become difficult to compare across versions. Reviewers also need a consistent way to flag unsupported citations and cases outside their assigned scope.

Delivery workflow

Run 4,200 evaluations a month against six rubric dimensions across three model versions. Calibrate reviewers on shared examples and send ambiguous cases to the assigned reviewer lead for adjudication.

Calibrate reviewers on a shared set of queries with known reference passages and annotated scoring examples. Score each dimension against the client’s rubric and attach evidence for a disputed or unsupported claim. Record the model version and rubric version before sending disagreements to the reviewer lead.

Keep adjudication decisions with the rationales and the references used by reviewers. Use them to explain recurring disagreements and update calibration materials after approval. Check agreement and evidence quality within individual dimensions, including examples that should trigger further review.

Delivery team

Poland, about 11 FTE. 1 eval lead, 2 QA reviewers and 8 calibrated reviewers.

Systems and tools

LangSmith, client eval platform, Google Sheets.

Client decision rights

The AI company owns rubric design, model selection, and deployment.

Early milestone

The week 4 record shows 86% agreement between reviewers.

Outcomes

Company-reported results, with the baseline, measurement window and supporting record described for each measure.

Reported results: Human evaluation for legal language models
MeasureBaselineResultWindowSupporting record
Inter-rater agreement78%94%4 monthsEvaluation platform export
Evaluation throughput900 / month4,200 / month4 monthsEvaluation platform export
Hallucination detection rate71%93%4 monthsEvaluation platform export

Interpreting the results

The increase from 78% to 94% inter-rater agreement is meaningful with the same agreement method and an appropriately difficult shared set. Measure hallucination detection against responses with known unsupported claims and report the sample size. Compare model versions on a common query set and stable rubric.

Recommendations for new clients

For teams commissioning human evaluation, test reproducibility on a shared labeled set. Report the agreement method, rubric version, and how reference hallucinations were identified.

Start with the legal use cases the product covers most often and a set of contentious edge cases. Provide the approved reference material, rubric, sample scores, and adjudication owner. Ask reviewers to score the same items independently and use the disagreements to improve the rubric and onboarding.

Scope this workflow for your operation

Use the documented workflow, client-held decisions and operational checks to outline a pilot for your own workload.