Interrupt the run on both sides of a commit
Recovery tests need precise fault locations. Stopping a process at a random time rarely proves how it handles the gap between an external effect and its local record.
Read articleCobnex blogs / Global
Cobnex blog
AI implementation, software architecture and cloud operations for teams worldwide.
146 articles in AI engineering
Page 3 of 9
Find the development example hiding in the test set
Near-duplicate cases can make an evaluation look independent when it is testing familiar material. Inspect relationships between examples, not just identical strings.
Read articleUse a fluent response with the wrong extracted field
Schema checks will accept many semantically wrong values. Test a document where nearby fields are easy to confuse and assert the business meaning of each output.
Read articleTrigger an error and inspect where its payload goes
Error paths often log more than successful requests. Use a synthetic marker to find diagnostic copies and unexpected fallbacks without exposing real information.
Read articleMake the cheap model retry until it costs more
A lower initial request cost can disappear through repair loops. Test a persistent validation failure and verify that the task stops within its total allowance.
Read articleAsk a familiar question the corpus cannot answer
A model may know a plausible answer from general knowledge while the product requires approved evidence. Test that distinction with a question just outside the source collection.
Read articleCount forbidden passages before scoring relevance
Search quality and access safety answer different questions. Measure both so a useful-looking answer cannot hide a failed permission check.
Read articleSeparate candidate recall from final ranking quality
A missing document and a badly ordered document need different fixes. Measure the retrieval stages separately before changing the model or increasing the context window.
Read articleMeasure correct passages interpreted the wrong way
Retrieval can find the right words and still produce the wrong conclusion. Add interpretation checks for the context that document splitting can remove.
Read articleScore claim support separately from polished writing
A fluent answer can still misstate its evidence. Measure whether its claims are supported, whether important conditions survived and whether readers can inspect the source.
Read articleMeasure freshness from the business change
A healthy ingestion job can coexist with stale answers. Measure the interval a reader experiences, then use pipeline timings to explain it.
Read articleDistinguish suspicious output from a successful attack
An assistant can repeat hostile text without executing it, or claim to refuse while a tool still runs. Security results need observable application outcomes.
Read articleCount attempted actions as well as completed tasks
Task completion does not show how much unnecessary or prohibited work an agent tried along the way. Measure the action path as well as the final result.
Read articleMeasure approvals that actually change the outcome
A high approval rate does not show that review is useful. Look at corrections, missed errors and the time people need to understand a proposal.
Read articleMeasure recovery by the final business state
A restarted worker is not a recovered task until its effects are correct and its status is understandable. Measure duplication, unresolved work and operator effort together.
Read articleAn average score can hide the expensive failure
Keep task coverage and failure consequence visible. A candidate that improves easy answers while mishandling a critical exception may be a worse release.
Read articleCompare candidate and baseline on identical tasks
A fair release comparison holds the task and evidence steady, then examines changed outcomes. Separate quality, useful completion and operational cost.
Read articleFalse acceptance matters more than coverage alone
Processing more documents automatically is useful only if the accepted records are reliable. Measure incorrect acceptance separately from the proportion sent to review.
Read article