This lesson's interactive features are locked, please to keep using them
Error Analysis
When search fails, don't just shrug. Figure out why! Be specific.
Metrics tell you what is wrong. Error analysis tells you why it's wrong.
Debugging the Pipeline
Let's say your system doesn't return documents that should match (i.e., false negatives). For example:
- User searches: "bear movie"
- Expected: "The Revenant," "Paddington"
- Actual: "The Revenant" (missing "Paddington")
With these types of issues, you'll need to debug the entire search pipeline to find out where the problem lies.
- Preprocessing
- Query rewriting
- Keyword search
- Semantic search
- Re-ranking
I recommend adding optional debug logging at each stage so that it's easy to see where results are lost or altered.
Homing in on the Problem
Once you find the part of the pipeline that's failing, it's a lot easier to fix. But sometimes, you need to dig even deeper.
For example, you might find that your system should handle both short and long queries, but it often fails differently based on length:
- "bear" → Too ambiguous; returns random bear content
- "1980s animated family-friendly bear adventure movie" → Too specific; no exact matches despite relevant content existing
To debug this, I'd keep the query intent the same, but vary the length of the input queries and see how the system responds. In this example I would likely be focused specifically on the query rewriting part of the pipeline.