Case study · AI data operations

Prompt regression testing for AI software

Company-reported case study

Actigy BPO ran a repeatable evaluation queue for PromptWorks’ weekly prompt releases. Reported regression detection moved from 61% to 94% over three months; prompt, AI-model and release decisions stayed with the product team.

Actigy BPO confirms the engagement and supporting records. Results apply to the stated scope and measurement windows.

Regression detection rate

Result

94%

Baseline: 61%

Window: 3 months

Supporting record: Evaluation test run

Output quality regressions

Result

4% of releases

Baseline: 28% of releases

Window: 3 months

Supporting record: Evaluation test run

Test coverage

Result

1,200 cases

Baseline: 180 cases

Window: 3 months

Supporting record: Airtable test inventory

Client profile

PromptWorks, a US AI writing software company with 45 staff.

Challenge and scope

Weekly changes ship without full regression coverage. The baseline has quality regressions in 28% of releases.

A prompt edit can improve a frequently tested output while producing a regression in a less common workflow. Model provider changes and updates to retrieval context can also affect results. Without recording those inputs, a product team can spend time trying to reproduce a defect against a different setup from the original test.

Delivery workflow

Run up to 1,200 test cases against 340 prompt variants across three model providers. Attach examples of failures and defer prompt, model, and release decisions to the product team.

Select baseline cases from the stable test set and record the prompt version, model settings, input, and expected behavior. Run the same cases on the candidate versions and retain the failed outputs. Group failures by task, trigger, and severity so the release owner can decide what needs action.

Maintain a set of known failures alongside broadly representative examples and add new tests after approved production incidents. Record cases requiring manual adjudication and make reruns reproducible with the applicable identifiers. Route high-impact failures to the product team with the relevant baseline and candidate outputs.

Delivery team

Poland, about 7 FTE. 1 QA lead, 2 prompt testers and 4 evaluation associates.

Systems and tools

PromptLayer, client eval harness, Airtable.

Client decision rights

The product team owns prompt design, model selection, and release approval.

Early milestone

At week 3, the regression detection rate is 78%.

Outcomes

Company-reported results, with the baseline, measurement window and supporting record described for each measure.

Reported results: Prompt regression testing for AI software
MeasureBaselineResultWindowSupporting record
Regression detection rate61%94%3 monthsEvaluation test run
Output quality regressions28% of releases4% of releases3 monthsEvaluation test run
Test coverage180 cases1,200 cases3 monthsAirtable test inventory

Interpreting the results

Read detection rising from 61% to 94% on the defined regression set with the share of released changes that later regress. More test cases contribute to coverage when they reach distinct workflows and known failure types. Keep reporting aligned to the release and model versions used for each comparison.

Recommendations for new clients

For AI software teams releasing prompts weekly, record the prompt and model versions and test the harness against known failures. Compare results using a stable, representative test set.

Bring prior prompt changes and examples of output regressions, including failures that appeared only with one model provider. Agree a base suite and an owner for test expectations. Run it on a known historical change and check whether failures are reproducible and sufficiently specific for the release decision.

Scope this workflow for your operation

Use the documented workflow, client-held decisions and operational checks to outline a pilot for your own workload.