Client profile
PromptWorks, a US AI writing software company with 45 staff.
Challenge and scope
Weekly changes ship without full regression coverage. The baseline has quality regressions in 28% of releases.
A prompt edit can improve a frequently tested output while producing a regression in a less common workflow. Model provider changes and updates to retrieval context can also affect results. Without recording those inputs, a product team can spend time trying to reproduce a defect against a different setup from the original test.
Delivery workflow
Run up to 1,200 test cases against 340 prompt variants across three model providers. Attach examples of failures and defer prompt, model, and release decisions to the product team.
Select baseline cases from the stable test set and record the prompt version, model settings, input, and expected behavior. Run the same cases on the candidate versions and retain the failed outputs. Group failures by task, trigger, and severity so the release owner can decide what needs action.
Maintain a set of known failures alongside broadly representative examples and add new tests after approved production incidents. Record cases requiring manual adjudication and make reruns reproducible with the applicable identifiers. Route high-impact failures to the product team with the relevant baseline and candidate outputs.
Delivery team
Poland, about 7 FTE. 1 QA lead, 2 prompt testers and 4 evaluation associates.
Systems and tools
PromptLayer, client eval harness, Airtable.
Client decision rights
The product team owns prompt design, model selection, and release approval.
Early milestone
At week 3, the regression detection rate is 78%.
Outcomes
Company-reported results, with the baseline, measurement window and supporting record described for each measure.
| Measure | Baseline | Result | Window | Supporting record |
|---|---|---|---|---|
| Regression detection rate | 61% | 94% | 3 months | Evaluation test run |
| Output quality regressions | 28% of releases | 4% of releases | 3 months | Evaluation test run |
| Test coverage | 180 cases | 1,200 cases | 3 months | Airtable test inventory |
Interpreting the results
Read detection rising from 61% to 94% on the defined regression set with the share of released changes that later regress. More test cases contribute to coverage when they reach distinct workflows and known failure types. Keep reporting aligned to the release and model versions used for each comparison.
Recommendations for new clients
For AI software teams releasing prompts weekly, record the prompt and model versions and test the harness against known failures. Compare results using a stable, representative test set.
Bring prior prompt changes and examples of output regressions, including failures that appeared only with one model provider. Agree a base suite and an owner for test expectations. Run it on a known historical change and check whether failures are reproducible and sufficiently specific for the release decision.