This lesson's interactive features are locked, please to keep using them

RSS

The whole point of the gator program is to fetch the RSS feed of a website and store its content in a structured format in our database. That way we can display it nicely in our CLI.

RSS stands for "Really Simple Syndication" and is a way to get the latest content from a website in a structured format. It's fairly ubiquitous on the web: most content sites have an RSS feed.

Structure of an RSS Feed

RSS is a specific structure of XML (I know, gross). We will keep it simple and only worry about a few fields. Here's an example of the documents we'll parse:

<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
  <title>RSS Feed Example</title>
  <link>https://www.example.com</link>
  <description>This is an example RSS feed</description>
  <item>
    <title>First Article</title>
    <link>https://www.example.com/article1</link>
    <description>This is the content of the first article.</description>
    <pubDate>Mon, 06 Sep 2021 12:00:00 GMT</pubDate>
  </item>
  <item>
    <title>Second Article</title>
    <link>https://www.example.com/article2</link>
    <description>Here's the content of the second article.</description>
    <pubDate>Tue, 07 Sep 2021 14:30:00 GMT</pubDate>
  </item>
</channel>
</rss>

We'll then parse this kind of document into a JavaScript objects like this:

type RSSFeed = {
  channel: {
    title: string;
    link: string;
    description: string;
    item: RSSItem[];
  };
};

type RSSItem = {
  title: string;
  link: string;
  description: string;
  pubDate: string;
};

If our program were running in a browser, we could use the built-in DOMParser API, but since we're in Node.js, we'll use fast-xml-parser instead.

Assignment

Install the fast-xml-parser package and its types:

npm i fast-xml-parser

Write a fetchFeed(feedURL: string) function. It should fetch a feed from the given URL, and, assuming that nothing goes wrong, return a filled-out RSSFeed object.
1. Fetch the feed data:
  1. I set the User-Agent header to gator This is a common practice to identify your program to the server.
  2. I resolved the response to a string using .text()
2. Parse the XML:
  1. Use the XMLParser constructor from the fast-xml-parser package to create a new parser object. Set processEntities to false so the current feed parses cleanly.
  2. Use the parse() method on the parser object to convert the XML string into a JavaScript object.
3. Extract the channel field:
  1. Verify that the channel field exists before proceeding. Handle errors if it does not.
4. Extract metadata:
  1. Ensure these fields are present and valid before continuing.
  2. Retrieve the title, link, and description fields from the channel field.
5. Extract feed items:
  1. Create an items array.
    1. If the channel field has an item field, it should be either an object or an array of objects. Use the Array.isArray() function to verify this and set the items array accordingly.
    2. Otherwise, set the item field to an empty array.
  2. For each item:
    1. Extract fields such as title, link, description, and pubDate.
    2. Skip any item that has missing or invalid fields.
6. Assemble the result:
  1. Create an object with the channel metadata and the list of items.
  2. Return this object as the final output.
Add an agg command. Later this will be our long-running aggregator service. For now, we'll just use it to fetch a single feed and ensure our parsing works. It should fetch the feed found at https://www.wagslane.dev/index.xml and print the entire object to the console.

Run and submit the CLI tests.