This lesson's interactive features are locked, please to keep using them

Adding Recursion

This is going to be the largest step so far, and will require some "figuring things out on your own"... you got this.

Assignment

Create a crawl_page function that uses recursion to traverse a website. Recursion is useful because we want to continue to crawl each URL we find on a page until we've crawled every page on the site. Here's my function signature:

def crawl_page(base_url, current_url=None, page_data=None):

base_url is the root URL of the website we're crawling
current_url is the current URL we're crawling

In the first call to crawl_page() current_url is a copy of base_url, but as we make further HTTP requests to all the URLs we find on the base_url, the current_url value will change while the base stays the same.

The page_data dictionary stores all the rich data we've extracted from each page, keyed by normalized URL. This function should continue to pass the same dictionary to itself.

Here's my pseudocode:

Make sure the current_url is on the same domain as the base_url. If it's not, just return. We don't want to crawl the entire internet, just the domain in question.
Get a normalized version of the current_url.
Check if we've already crawled this page by checking if the normalized URL is already a key in the page_data dictionary. If we have, just return - we don't want to crawl the same page twice.
Get the HTML from the current URL, and add a print statement so you can watch your crawler in real-time.
Assuming all went well with the request, use extract_page_data() to get the rich data from this page and add it to the page_data dictionary using the normalized URL as the key.
Get all the URLs from the response body HTML
Recursively crawl each URL on the page

Be careful testing this! Be sure to add print statements so you can see what your crawler is doing, and kill it with Ctrl+C if it's stuck in a loop. If you make too many spammy requests to a website (including the sandbox) you could get your IP address blocked.

Call crawl_page in the main function instead of get_html. When it's complete, print some information about the collected data to the console - like the number of pages found and iterate over the dictionary values to show the data.
Test your program by running it against https://learnwebscraping.dev/practice/ecommerce/.

uv run main.py https://learnwebscraping.dev/practice/ecommerce/

When you're satisfied that everything is working, you can move on.