This lesson's interactive features are locked, please to keep using them

Stemming

Now that we have a list of tokens, we need to reduce related word forms to a shared base form. This helps match different variations of the same word.

Examples of Word Variations

running, runs → run
jumping, jumps, jumped → jump
watching, watches, watched → watch

These variations are normalized to their stem, a simplified form used for matching. The process is called stemming.

Why Stem Words?

Our goal is to match words from the user's input to words in our dataset. If we don't stem words, we can miss valid results. Here's an example:

User query: "running"

A River Runs Through It: Not returned – no exact match for "running"
Run Baby Run: Not returned – no exact match for "running"

Ugh! Both contain the concept, but in different forms that don't quite match the query.

Assignment

Implementing stemming from scratch is a lot of work, so we'll use the nltk.stem library to handle it for us.

Install the nltk library:
```
uv add nltk==3.9.1
```
Import the PorterStemmer class from nltk.stem:
```
from nltk.stem import PorterStemmer
```
Create an instance of PorterStemmer:
```
stemmer = PorterStemmer()
```
After removing stop words from the tokens, reduce each token to its stem before matching. Use stemmer.stem(token) to get the stemmed form of each token.

Run and submit the CLI tests.