

0 / 2 embers
0 / 3000 xp
click for more info
Complete a lesson to start your streak
click for more info
Difficulty: 5
click for more info
Not enough gems
Cost: 6 gems
1: Welcome
incomplete
2: Python Setup
incomplete
3: Normalize URLs
incomplete
4: Extract Page Content
incomplete
5: Extract Links and Images
incomplete
6: Structure Page Data
incomplete
This lesson's interactive features are locked, please to keep using them
We need to extract links from the HTML both the links people can click AND the images that are displayed. This gives us a complete picture of what resources each page references.
For example, this HTML page has both a link and an image:
<html>
<body>
<a href="https://crawler-test.com">Go to Boot.dev</a>
<img src="/logo.png" alt="Boot.dev Logo" />
</body>
</html>We need to extract both https://crawler-test.com (from the link) and /logo.png (from the image).
def get_urls_from_html(html, base_url):html is an HTML string
base_url is the root URL of the website we're crawling. This will allow us to rewrite relative URLs into absolute URLs.
It returns an un-normalized list of all the URLs found within the HTML.
In your tests, make sure that:
relative URLs are converted to absolute URLs
you find all the <a> tags in a body of HTML
Here's one example test case to give you an idea:
def test_get_urls_from_html_absolute(self):
input_url = "https://crawler-test.com"
input_body = '<html><body><a href="https://crawler-test.com"><span>Boot.dev</span></a></body></html>'
actual = get_urls_from_html(input_body, input_url)
expected = ["https://crawler-test.com"]
self.assertEqual(actual, expected)find_all() looks through a tag's descendants and returns the elements that match your filters.
You can use the .get() method to retrieve the value of a specific attribute.
urljoin() intelligently combines a "base URL" with a relative URL.
In HTML, "anchor" elements are links. With their href attribute containing the actual URL. e.g:
<a href="https://www.boot.dev">Learn Backend Development</a>def get_images_from_html(html, base_url):html is an HTML string
base_url is the root URL of the website we're crawling. This will allow us to rewrite relative URLs into absolute URLs.
It returns an un-normalized list of all the image URLs found within the HTML.
In your tests, make sure:
relative URLs are converted to absolute URLs
Handle cases where attributes might be missing
Here's one example test case to give you an idea:
def test_get_images_from_html_relative(self):
input_url = "https://crawler-test.com"
input_body = '<html><body><img src="/logo.png" alt="Logo"></body></html>'
actual = get_images_from_html(input_body, input_url)
expected = ["https://crawler-test.com/logo.png"]
self.assertEqual(actual, expected)Run and submit the CLI tests from the root of your module.