A search result is useful only in the context of a question

Separate finding a known item from exploring a topic

Someone searching for an exact invoice reference expects one specific record. Someone searching for “recover a failed deployment” may need several related procedures. These tasks need different evidence of success.

Create query groups for known-item lookup, topic discovery, abbreviations, misspellings and questions with no appropriate result. Include the audience's language, such as internal product names and the terms customers actually use.

Do not build the entire test set by copying document titles. That makes exact matching look excellent while avoiding the vocabulary gap that caused the search problem in the first place.

Judge documents against intent

For each query, write a short explanation of what the person needs. Have a knowledgeable reviewer rate candidate documents against that intent, using a clear scale and recording disagreements.

A document can mention the query words without solving the task. An old deployment procedure may be topically relevant but unsafe for the current platform. Decide whether freshness and applicability belong in eligibility filters, relevance judgments or both.

Keep permission checks separate from relevance scoring. An unauthorised document is not a low-quality result to rank lower. It must be excluded under the application's access rules.

Use metrics that match the task

For a known-item query, the position of the first relevant result is useful. For broader discovery, the quality of several top results matters. Elastic's ranking evaluation API supports established metrics using rated documents, but choosing a metric remains a product decision.

An illustrative result list contains relevant documents at positions one and five. A first-result measure can look excellent even though the intervening results are distracting. A top-five precision measure sees two useful results out of five. Neither number alone explains every search experience.

Inspect per-query changes as well as aggregate scores. Improving many easy queries can hide a serious regression in a critical support query.

Change one meaningful variable at a time

Start with a baseline and a versioned corpus. Test a synonym rule, field weighting or reranking change against the same judgments where possible. Record index and query configuration so the result can be reproduced.

Search scores are model-specific signals, not universal probabilities. Azure's BM25 documentation describes an unbounded score and notes variation with index statistics. A threshold copied from one query or index is not automatically meaningful elsewhere.

Confirm that the interface completes the retrieval work

A correctly ranked document can still fail the user if its title is vague, its snippet hides the relevant section or the destination is broken. Test opening the result and finding the answer.

Use live behaviour as additional evidence, while recognising that clicks are influenced by position and presentation. A click is a clue, not a human relevance judgment. The strongest evaluation combines a maintained test set with observation of completed tasks.