

0 / 2 embers
0 / 3000 xp
click for more info
Complete a lesson to start your streak
click for more info
Difficulty: 4
click for more info
Not enough gems
Cost: 6 gems
1: Welcome
incomplete
2: Python Setup
incomplete
3: Normalize URLs
incomplete
4: Extract Page Content
incomplete
5: Extract Links and Images
incomplete
6: Structure Page Data
incomplete
This lesson's interactive features are locked, please to keep using them
In this project, we'll build a Web Crawler in Python! To rank well in Google Search, websites need to internally link pages one to another. For example, a blog post about the benefits of haircuts should probably link to my post about the best places to get haircuts.
We're going to write a Python CLI application that generates an "internal links" report for any website on the internet by crawling each page of the site.
If you try to complete this course without WSL 2 installed on Windows, you're gonna have a bad time.
Get hands-on practice with local Python development and tooling
Practice making HTTP requests in Python
Learn how to parse HTML
Practice unit testing
Before we dive into the project, let's make sure you are all set up properly. You will need:
Python 3.10+ installed (see the bookbot project for help if you don't already have it)
uv project and package manager
If you're using VS Code, I recommend installing the Python extension. It's not required, but it makes working with Python a lot easier.
You will need the Boot.dev CLI installed, and you'll need to be logged in.
Type python3 --version to make sure that you have Python installed.
Also, confirm that you have uv installed by typing uv --version.
Submit the CLI tests. There's no penalty on failure for this lesson.