This lesson's interactive features are locked, please to keep using them

Tokenization

Tokenization means splitting text into smaller pieces, called tokens. Simple word-based tokenization looks like this:

Input: "The Matrix is a great movie!"
Output: ["the", "matrix", "is", "a", "great", "movie"]

Many tokenization methods exist, but we'll use word-based tokenization for keyword search. We already handled case insensitivity and punctuation removal in the previous steps.

You might hear that "tokens" in the context of LLMs don't map one-to-one with words, and that's true, so don't let that trip you up here.

Assignment

We want Webflyx users to be able to search for partial matches, not just exact full-query substrings. For example, the query "Great Bear" should match the title "Big Bear" because the bear token appears in both.

Wrap this logic in a helper function! You'll need to apply the exact same processing throughout the course.

Split the query and the title into tokens.
- .split() on whitespace
- Remove any empty tokens
  
  Call .split() without an argument. Passing " " only splits literal spaces, not all whitespace.
Update your matching logic to allow matches where at least one query token appears in any title token (stemming not required).

Run and submit the CLI tests.