This lesson's interactive features are locked, please to keep using them

Extract Links and Images

We need to extract links from the HTML both the links people can click AND the images that are displayed. This gives us a complete picture of what resources each page references.

For example, this HTML page has both a link and an image:

<html>
  <body>
    <a href="https://crawler-test.com">Go to Boot.dev</a>
    <img src="/logo.png" alt="Boot.dev Logo" />
  </body>
</html>

We need to extract both https://crawler-test.com (from the link) and /logo.png (from the image).

Assignment

Write tests for a new function called get_urls_from_html. Here's the signature:

def get_urls_from_html(html, base_url):

html is an HTML string
base_url is the root URL of the website we're crawling. This will allow us to rewrite relative URLs into absolute URLs.
It returns an un-normalized list of all the URLs found within the HTML, and an error if one occurs.

In your tests, make sure that:

relative URLs are converted to absolute URLs
you find all the <a> tags in a body of HTML

Here's one example test case to give you an idea:

def test_get_urls_from_html_absolute(self):
    input_url = "https://crawler-test.com"
    input_body = '<html><body><a href="https://crawler-test.com"><span>Boot.dev</span></a></body></html>'
    actual = get_urls_from_html(input_body, input_url)
    expected = ["https://crawler-test.com"]
    self.assertEqual(actual, expected)

Implement the get_urls_from_html function. Continue using the Beautiful Soup library. Here are a few hints you may find useful:

find_all() looks through a tag's descendants and returns the elements that match your filters.
You can use the .get() method to retrieve the value of a specific attribute.
urljoin() intelligently combines a "base URL" with a relative URL.
In HTML, "anchor" elements are links. With their href attribute containing the actual URL. e.g:

<a href="https://www.boot.dev">Learn Backend Development</a>

Write tests for a new function called get_images_from_html. Here's the signature:

def get_images_from_html(html, base_url):

html is an HTML string
base_url is the root URL of the website we're crawling. This will allow us to rewrite relative URLs into absolute URLs.
It returns an un-normalized list of all the image URLs found within the HTML, and an error if one occurs.

In your tests, make sure:

relative URLs are converted to absolute URLs
Handle cases where attributes might be missing

Here's one example test case to give you an idea:

def test_get_images_from_html_relative(self):
    input_url = "https://crawler-test.com"
    input_body = '<html><body><img src="/logo.png" alt="Logo"></body></html>'
    actual = get_images_from_html(input_body, input_url)
    expected = ["https://crawler-test.com/logo.png"]
    self.assertEqual(actual, expected)

Implement the get_images_from_html function.
Make sure you have at least 3 test cases for each function you're testing.

Run and submit the CLI tests from the root of your module.