

0 / 2 embers
0 / 3000 xp
click for more info
Complete a lesson to start your streak
click for more info
Difficulty: 5
click for more info
Not enough gems
Cost: 6 gems
1: Coroutines and Context Managers
incomplete
2: Concurrency
incomplete
3: Max Pages
incomplete
This lesson's interactive features are locked, please to keep using them
Your web crawler works – but it's crawling pages one at a time. It would take us a really long time to crawl a large website. Let's make it faster using coroutines. This is another long step, but don't get discouraged!
base_url (the starting URL)
base_domain (the domain name)
page_data (our dictionary of page data, keyed by normalized URL)
visited (a set of normalized URLs already scheduled for crawling)
lock (an asyncio.Lock to safely update page_data)
max_concurrency (to limit the number of requests allowed at once)
semaphore (an asyncio.Semaphore - pass it the value of max_concurrency)
session (an aiohttp.ClientSession for making HTTP requests)
async def __aenter__(self):
self.session = aiohttp.ClientSession()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
await self.session.close()async def add_page_visit(self, normalized_url):Take a normalized URL
Use async with on the lock field to safely check and update visited
If the normalized URL is already in visited, return False
Otherwise, add it to visited and return True before fetching the page
Use the aiohttp client you set in the session field along with async with to fetch the page.
Keep the same error, status, and header checks
Keep the same-domain check: if the current URL's domain differs from base_domain, return before recording a visit or fetching the page
Call and await your new add_page_visit method. If it isn't a new page, return early
Use async with self.semaphore to limit the number of concurrent requests
Fetch the page's HTML and extract page data using extract_page_data
Add the page data to page_data dictionary using the normalized URL as the key (use the lock to do this safely)
Use the extracted page data's outgoing_links as the URLs to crawl next
For each URL, create a task to crawl it using asyncio.create_task
Wait for all tasks with await asyncio.gather(*tasks)
Creates and uses an AsyncCrawler in an async with block
Calls/awaits crawl() and returns the final page_data dictionary
Instead of calling crawl_page, call crawl_site_async(base_url) and await the results
When iterating over the page data, use page_data.values() to get the page dictionaries
Update the if __name__ == "__main__" block to run your async main function using asyncio.run(main())
Make sure you're not crawling the same page multiple times. That's why my add_page_visit method returns a boolean: to indicate if it's the first time we've seen the page.
Make sure you're using the semaphore to cap simultaneous requests