Client profile
Radiant Health AI, a US radiology AI company with 60 staff.
Challenge and scope
The radiology AI team needs bounding boxes for 14 pathology classes and has limited clinical reviewer capacity.
Different imaging modalities can make the same finding look different, and rare classes have fewer examples for calibration. Annotators may be uncertain about the correct bounding region or class. The workflow needs to capture that uncertainty so an approved clinical reviewer can resolve it in the client’s environment.
Delivery workflow
Run bounding-box annotation in the approved viewer and annotation platform. Have quality reviewers check labels by modality and escalate ambiguous findings for a client clinical decision.
Assign studies by modality and case type and use the client-provided examples for drawing and labeling boxes. Track study and image references in the approved viewer. Flag ambiguous findings or visibility problems for clinical review and record the specific labeling question for that reviewer.
Sample accepted labels by modality and class and track disagreements separately. Record box and class corrections with the relevant review decision so rules can be clarified and affected batches can be checked. Give the client a list of unresolved labels and the volume of accepted studies.
Delivery team
Poland, about 12 FTE. 1 clinical annotation lead, 2 QA radiographers and 9 annotation specialists.
Systems and tools
Client annotation platform, OHIF viewer, internal QA console.
Client decision rights
The AI company owns clinical ground truth, model validation, and regulatory submissions.
Early milestone
At week 6, the annotation accuracy reaches 98.1%.
Outcomes
Company-reported results, with the baseline, measurement window and supporting record described for each measure.
| Measure | Baseline | Result | Window | Supporting record |
|---|---|---|---|---|
| Annotation accuracy | 96.4% | 99.6% | 6 months | QA review sample |
| Throughput | 1× | 6.2× | 6 months | Annotation platform export |
| Inter-annotator agreement | 82% | 95% | 6 months | QA review sample |
Interpreting the results
The 99.6% annotation accuracy and 95% inter-annotator agreement measure different qualities. Define what makes a box and class acceptable and record clinical adjudication for disagreements. Compare accepted studies per period alongside error rates and include separate results for rare classes.
Recommendations for new clients
For imaging AI teams, establish an approved reference set with clinical reviewers. Measure annotation quality by modality and class, with a separate sample for rare findings.
Use a client-approved reference set with common and rare findings across modalities. Agree the review access, what should be marked as ambiguous, and who can resolve a clinical labeling question. Have the client review the first batch and the disagreement log before assigning the next group of studies.