This lesson's interactive features are locked, please to keep using them

Extract Page Content

Our web crawler will need to know how to read a page of HTML. Now that we can normalize URLs, let's start extracting actual content from web pages. A web scraper that only normalizes URLs isn't very useful - we need to parse HTML and extract the meaningful information.

Let's start with just parsing the <h1> and first <p> tags.

For example, from this HTML page:

<html>
  <body>
    <h1>Welcome to Boot.dev</h1>
    <main>
      <p>Learn to code by building real projects.</p>
      <p>This is the second paragraph.</p>
    </main>
  </body>
</html>

We want to extract:

<h1>: "Welcome to Boot.dev"
The first <p>: "Learn to code by building real projects."

Assignment

Write tests for a new function called get_heading_from_html. Here's the signature:

def get_heading_from_html(html: str) -> str:

html is an HTML string
In your tests make sure that the function returns the text content of the <h1> tag if present, or the <h2> tag as a fallback.
Returns an empty string if neither an <h1> nor an <h2> tag is found.

Implement the get_heading_from_html function. We'll be using the Beautiful Soup library. We've already installed it as a dependency, you can import it using:

from bs4 import BeautifulSoup, Tag

If you're using type hints and a static checker, it can help to narrow the result of .find() before calling .get_text():

return h_tag.get_text(strip=True) if isinstance(h_tag, Tag) else ""

I'll try not to give too many hints: read the package docs linked above, especially the Quick Start. Reading docs is vital practice, that said, here are a few hints:

.find() scans the document looking for the passed in tag. Limited to 1.
get_text() Extracts only the human readable text beneath the tag as a single unicode string.

Write tests a new function called get_first_paragraph_from_html. Here's the signature:

def get_first_paragraph_from_html(html: str) -> str:

In your tests make sure that the tests return the text content of the first <p> tag.
Returns an empty string if no <p> tag is found.

You may find that the first <p> tag doesn't always have the best or most useful results. I'd recommend searching for the <main> tag if it exists and find the first <p> tag within it, if it doesn't exist fallback to just
the first <p> tag.

Here are some example test cases to get you started:

def test_get_heading_from_html_basic(self):
    input_body = "<html><body><h1>Test Title</h1></body></html>"
    actual = get_heading_from_html(input_body)
    expected = "Test Title"
    self.assertEqual(actual, expected)


def test_get_first_paragraph_from_html_main_priority(self):
    input_body = """<html><body>
        <p>Outside paragraph.</p>
        <main>
            <p>Main paragraph.</p>
        </main>
    </body></html>"""
    actual = get_first_paragraph_from_html(input_body)
    expected = "Main paragraph."
    self.assertEqual(actual, expected)

Make sure you have at least 3 test cases for each function you are testing!

Run and submit the CLI tests from the root of your module.