This lesson's interactive features are locked, please to keep using them

Get HTML

Let's start crawling real web pages! For these remaining steps, use https://learnwebscraping.dev/practice/ecommerce/. A single product page like https://learnwebscraping.dev/practice/ecommerce/products/ashenfang-longsword-fan-1001/ is a good place to start.

Assignment

Create a get_html function that takes a URL string and returns an HTML string or raises an exception

def get_html(url):

For now, your function should:

Import and use the requests package we installed to fetch the webpage of the url
Set a User-Agent header (e.g. BootCrawler/1.0) to avoid being blocked by servers.

requests.get(url, headers={"User-Agent": "BootCrawler/1.0"})

Raise an error if the HTTP status code is an error-level code (400+)
Raise an error if the response content-type header is not text/html
Raise any other possible errors
Return the webpage's HTML if successful

I'd argue that it's not necessary to create unit tests for a function like get_html. It's primarily just side effects (internet access), making it not a pure function like normalize_url and get_urls_from_html. Most of the "logic" in get_html is just a couple of standard library function calls: and there's not much reason to test the standard library.

Call get_html from your main function and print the result. It should print some HTML that it fetched from the internet!

Run and submit the CLI tests.

Notice that they're grabbing some HTML from the main page of Wikipedia.