Stop Words

• bear
• panda
• london
• protector

• a
• the
• is
• of
• in

These are examples of stop words: common words that usually don't add much meaning. In short, we're going to remove them.

Our goal is to match words from the user's input to words in our dataset. If we don't remove stop words, we can get some pretty misleading results. Here's an example:

User query: "the bear"

Results:

• "Jungle Bear": matches "bear".
• "The Terminator": matches "the".

Oof! Both movies match, but "Jungle Bear" is clearly more relevant. "The Terminator" matches only on a low-value stop word.

Remove stop word tokens from both the user query and the title tokens.

1. Create a new data/stopwords.txt file and paste the words at the end of this page into it.
2. Read the stop words from the file and store them in a list.
   • .read() the file
   • .splitlines() to split it into a list of words
3. Preprocess the stop words the same way you preprocess the user query and title text.
   • For example, aren't should become arent before you compare tokens against it!
4. Load and preprocess the stop words once, then reuse that list whenever you filter tokens.
5. Filter any stop words out of both the user query tokens and title tokens before matching.

Run and submit the CLI tests.