Manual Evaluation

You can't improve a search system if you don't know how it's performing! But before we talk about metrics, let's talk about a critically important part: the vibe check.

Does your search feel good?

Metrics are great, but:

• Evaluation criteria are always imperfect.
• LLMs often cause unexpected results.
• You need to develop intuition about the kinds of ways your search can fail.

So manually testing your search is actually really important as a first step. I run test queries and ask myself the following questions about the results:

• What's here that shouldn't be?
• What's not here that should be?
• Would I click on these results?

That's a great starting point, but go even deeper and really think critically about the results. There are many "technically correct" results that are still not ideal. The correct search results depends on what you're looking for – there is no universal truth. Here's some scenarios to consider:

• If a user searched for "Gods Among Men," would "Thor" or "Mortal Kombat" be more likely to appeal to the average user?
• If a user searched for "Dinosaur," and the top 10 results were "The Land Before Time" 1–10, with "Jurassic Park" at #11, would that be a good result (even though it makes sense technically)?

And consider some additional factors:

• Did the result give you enough information to know whether the movies are relevant?
• Would it be better to return fewer results because the last few usually aren't relevant?
• Would it be better to return more results because there are more highly relevant options that just missed the cutoff?