This lesson's interactive features are locked, please to keep using them

Structure Page Data

So far we've built functions to help us normalize URLs and extract links/text from HTML. Now let's structure that data in a way that's much more usable.

Assignment

Write tests for a new function called extract_page_data. Here's the signature:

def extract_page_data(html: str, page_url: str):

html is an HTML string
page_url is the absolute URL of the page (used for converting relative URLs)
It returns a dictionary with keys: url, heading, first_paragraph, outgoing_links, image_urls

Here's one example test case to get you started:

def test_extract_page_data_basic(self):
    input_url = "https://crawler-test.com"
    input_body = """<html><body>
        <h1>Test Title</h1>
        <p>This is the first paragraph.</p>
        <a href="/link1">Link 1</a>
        <img src="/image1.jpg" alt="Image 1">
    </body></html>"""
    actual = extract_page_data(input_body, input_url)
    expected = {
        "url": "https://crawler-test.com",
        "heading": "Test Title",
        "first_paragraph": "This is the first paragraph.",
        "outgoing_links": ["https://crawler-test.com/link1"],
        "image_urls": ["https://crawler-test.com/image1.jpg"],
    }
    self.assertEqual(actual, expected)

Implement the extract_page_data function. You have already built all the logic for this function, it just needs to return the data you're already gathering in a dictionary!

If you want to type the returned dictionary shape, you can import TypedDict and define a PageData type:

from typing import TypedDict


class PageData(TypedDict):
    url: str
    heading: str
    first_paragraph: str
    outgoing_links: list[str]
    image_urls: list[str]

Run and submit the CLI tests to verify your extraction logic works correctly!