> ## Documentation Index
> Fetch the complete documentation index at: https://docs.booper.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Top HN post alerts

> Get alerted of top HN posts every hour

### Overview

Let's set up a script that alerts us every hour of new posts on the front page of Hacker News that have exceeded a certain point threshold.

Here's how we'll tackle it:

1. Write the script to fetch all front page posts with a minimum upvote score
2. Set up a notification to be sent to Discord whenever a new post is found
3. Configure the schedule so that it "remembers" which posts have already been sent through

<Accordion title="Just show me the code">
  See the finished code on [Replit](https://replit.com/@reichertjalex/hn).

  You can even use the Replit URL to run your job, as long as the Repl is running. Make sure to replace `YOUR_DISCORD_WEBHOOK_URL` with your [Discord webhook URL](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks), and then you can run the following in your terminal:

  <CodeGroup>
    ```bash cURL theme={null}
    # This assumes your API key is set in the current env
    # BOOPER_API_KEY=sk_...

    curl --location --request POST 'https://scheduler.booper.dev/api/jobs' \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $BOOPER_API_KEY" \
    --data-raw '{
        "method": "post",
        "url": "https://hn.reichertjalex.repl.co/api/run",
        "body": {
          "min": 200,
          "webhook": YOUR_DISCORD_WEBHOOK_URL,
          "exclude": "$previous.sent"
        },
        "repeat_every": [1, "hour"]
    }'
    ```

    ```bash CLI theme={null}
    # Assumes your webhook URL is set in the current env:
    # DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

    npx @booper/cli run \
      POST https://hn.reichertjalex.repl.co/api/run \
      --data 'min=200' \
      --data "webhook=$DISCORD_WEBHOOK_URL" \
      --data 'exclude=$previous.sent' \
      --every '1 hour'
    ```
  </CodeGroup>
</Accordion>

### Fetching top stories from HN

We'll use the [Hacker News API](https://github.com/HackerNews/API) endpoints to fetch the top page stories. Here's what that function might look like:

<CodeGroup>
  ```js JavaScript theme={null}
  const get = async (url) => fetch(url).then((r) => r.json());

  const fetchStories = async ({ pages = 1, min = 100, exclude = [] } = {}) => {
    const storyIds = await get(
      `https://hacker-news.firebaseio.com/v0/topstories.json`
    );

    return Promise.all(
      storyIds
        // Filter out posts we've already seen
        .filter((id) => !exclude.includes(id))
        // Limit to the first page (30 posts per page)
        .slice(0, pages * 30)
        .map(async (id) => {
          const story = await get(
            `https://hacker-news.firebaseio.com/v0/item/${id}.json`
          );

          return story;
        })
    );
  };
  ```

  ```ts TypeScript theme={null}
  const get = async (url: string) => fetch(url).then((r) => r.json());

  const fetchStories = async ({
    pages = 1,
    min = 100,
    exclude = [],
  }: {
    pages?: number;
    min?: number;
    exclude?: number[];
  }) => {
    const storyIds = await get(
      `https://hacker-news.firebaseio.com/v0/topstories.json`
    );

    return Promise.all(
      storyIds
        // Filter out posts we've already seen
        .filter((id: number) => !exclude.includes(id))
        // Limit to the first page (30 posts per page)
        .slice(0, pages * 30)
        .map(async (id: number) => {
          const story = await get(
            `https://hacker-news.firebaseio.com/v0/item/${id}.json`
          );

          return story;
        })
    );
  };
  ```
</CodeGroup>

It's basically an N+1 query, but no big deal. If we limit the results to the first page (i.e. the first 30 posts), the request is generally fast enough.

<Accordion title="View example output of HN posts" icon="code">
  Here's an example of what the [output](https://www.booper.dev/api/examples/hn?pages=1\&min=200) looks like:

  ```json theme={null}
  {
    "stories": [
      {
        "title": "Get a cable modem, go to jail (1999)",
        "url": "http://telecom.csail.mit.edu/judy-sammel.html",
        "score": 447
      },
      {
        "title": "Writing a C compiler in 500 lines of Python",
        "url": "https://vgel.me/posts/c500/",
        "score": 432
      },
      {
        "title": "Emacs Bedrock: A minimal Emacs starter kit",
        "url": "https://sr.ht/~ashton314/emacs-bedrock/",
        "score": 214
      }
      // ...
    ]
  }
  ```
</Accordion>

### Triggering a Discord alert when new posts are found

First, let's define two helper functions:

* `format`: formats story data into Discord markdown
* `notify`: sends an alert to Discord via webhook URL

<CodeGroup>
  ```js JavaScript theme={null}
  const format = (results) => {
    return results
      .map((p) => `[${p.score} points] [${p.title}](${p.url})`)
      .join("\n");
  };

  const notify = async (content, webhook = process.env.DISCORD_WEBHOOK_URL) => {
    return fetch(webhook, {
      headers: { "content-type": "application/json" },
      method: "POST",
      body: JSON.stringify({ content }),
    });
  };
  ```

  ```ts TypeScript theme={null}
  type Post = { id: number; score: number; title: string; url: string };

  const format = (results: Post[]) => {
    return results
      .map((p) => `[${p.score} points] [${p.title}](${p.url})`)
      .join("\n");
  };

  const notify = async (content: string, webhook = DISCORD_WEBHOOK_URL) => {
    return fetch(webhook, {
      headers: { "content-type": "application/json" },
      method: "POST",
      body: JSON.stringify({ content }),
    });
  };
  ```
</CodeGroup>

Now, the simplest thing we can do is just send an alert to Discord whenever we find posts that meet the minimum score requirement set in the request body.

Here we handle that in a [NextJS API endpoint](https://nextjs.org/docs/pages/building-your-application/routing/api-routes):

<CodeGroup>
  ```ts pages/api/hn.ts (NextJS) theme={null}
  export default async function handler(req, res) {
    const { min = 100, pages = 1 } = req.body;
    const stories = await fetchStories({ pages });
    // After we fetch the stories, we filter out the ones that
    // don't have the minimum score specified in the request body
    const results = stories.filter((s) => s.score >= min);

    if (results.length > 0) {
      // Send notification to Discord using the helpers defined above
      await notify(format(results));
    }

    return res.status(200).json({ results });
  }
  ```

  ```js app.js (NodeJS) theme={null}
  import express from "express";

  const app = express();

  app.use(express.json());

  app.post("/api/hn", async (req, res) => {
    const { min = 100, pages = 1 } = req.body;
    const stories = await fetchStories({ pages });
    // After we fetch the stories, we filter out the ones that
    // don't have the minimum score specified in the request body
    const results = stories.filter((s) => s.score >= min);

    if (results.length > 0) {
      // Send notification to Discord using the helpers defined above
      await notify(format(results));
    }

    return res.status(200).json({ results });
  });

  app.listen(process.env.PORT || 3000, () => console.log("Server initialized!"));
  ```
</CodeGroup>

This is fine if we run the script every hour or two, and don't mind seeing repeated links. But what if we want to avoid sending duplicates?

### Keeping track of alert history

There are two ways we can keep track of which posts have been sent out already.

When you create a job on a schedule, the scheduler will pass in some [metadata](/link-to-job-metadata) into the body of each request. Included in that metadata is the response from the previous run, which can be seen at `req.body.$previous`.

<Accordion title="Example using the `$previous` metadata" icon="code">
  Let's modify the API handler to take advantage of the `$previous` data:

  ```ts theme={null}
  export default async function handler(req, res) {
    const { min = 100, pages = 1 } = req.body;
    // Add this line! 👇
    const exclude = req.body.$previous?.sent || [];
    // And then pass in the `exclude` parameter to exclude
    // previously sent posts from the results of `fetchStories`
    const stories = await fetchStories({ pages, exclude });
    const results = stories.filter((s) => s.score >= min);

    if (results.length > 0) {
      await notify(format(results));
    }

    return res.status(200).json({
      // And modify the response to return all the IDs of sent posts,
      // including the previously sent posts (from `exclude`) 👇
      sent: [...exclude, ...results.map((s) => s.id)],
    });
  }
  ```

  If we want, we can also clean this up a bit by taking advantage of [dynamic values](/link-to-dynamic-values-in-job-metadata) in our job schedule configuration. When we set the request `body` for our job, we can write it like this:

  ```json theme={null}
  {
    // Set the minimum score requirement to 200 points
    "min": 200,
    // Pass in `req.body.$previous?.sent` as `req.body.exclude`
    "exclude": "$previous.sent"
  }
  ```

  If we do this, we can change the line above from this:

  ```ts theme={null}
  const exclude = req.body.$previous?.sent || [];
  ```

  To this:

  ```ts theme={null}
  const exclude = req.body.exclude || [];
  ```

  Now, assuming you've deployed your API endpoint to `https://yourdomain.com/api/hn`, you can create your scheduled job by running the following script in your terminal with your `BOOPER_API_KEY` set, and the `url` modified to the appropriate domain:

  <CodeGroup>
    ```bash cURL theme={null}
    # This assumes your API key is set in the current env
    # BOOPER_API_KEY=sk_...

    curl --location --request POST 'https://scheduler.booper.dev/api/jobs' \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $BOOPER_API_KEY" \
    --data-raw '{
        "method": "post",
        "url": "https://yourdomain.com/api/hn",
        "body": {"min": 200, "exclude": "$previous.sent"},
        "repeat_every": [1, "hour"]
    }'
    ```

    ```bash CLI theme={null}
    npx @booper/cli run \
      POST https://yourdomain.com/api/hn \
      --data 'min=200' \
      --data 'exclude=$previous.sent' \
      --every '1 hour'
    ```
  </CodeGroup>
</Accordion>

Also included in the metadata is the schedule's `state`, which can be accessed at `req.body.$state` and set or updated in the response.

<Accordion title="Example using the schedule `$state`" icon="code">
  Let's modify the API handler to take advantage of `$state`:

  ```ts theme={null}
  export default async function handler(req, res) {
    const { min = 100, pages = 1 } = req.body;
    // Add this line! 👇
    const exclude = req.body.$state?.sent || [];
    // And then pass in the `exclude` parameter to exclude
    // previously sent posts from the results of `fetchStories`
    const stories = await fetchStories({ pages, exclude });
    const results = stories.filter((s) => s.score >= min);

    if (results.length > 0) {
      await notify(format(results));
    }

    return res.status(200).json({
      $set: {
        sent: [...exclude, ...results.map((s) => s.id)],
      },
    });
  }
  ```

  This may look similar to what we did above when using the `$previous` metadata, but there is one key difference: **while `$previous` data is ephemeral, the schedule's `$state` is stored in the database**. This means that if a schedule is paused and then restarted again, the `$previous` data will be lost, but `$state` will be persisted.

  Once again, we can clean this up a bit by taking advantage of [dynamic values](/link-to-dynamic-values-in-job-metadata) in our job schedule configuration. When we set the request `body` for our job, we can write it like this:

  ```json theme={null}
  {
    // Set the minimum score requirement to 200 points
    "min": 200,
    // Pass in `req.body.$state?.sent` as `req.body.exclude`
    "exclude": "$state.sent"
  }
  ```

  If we do this, we can change the line above from this:

  ```ts theme={null}
  const exclude = req.body.$state?.sent || [];
  ```

  To this:

  ```ts theme={null}
  const exclude = req.body.exclude || [];
  ```

  Now, assuming you've deployed your API endpoint to `https://yourdomain.com/api/hn`, you can create your scheduled job by running the following script in your terminal with your `BOOPER_API_KEY` set, and the `url` modified to the appropriate domain:

  <CodeGroup>
    ```bash cURL theme={null}
    # This assumes your API key is set in the current env
    # BOOPER_API_KEY=sk_...

    curl --location --request POST 'https://scheduler.booper.dev/api/jobs' \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $BOOPER_API_KEY" \
    --data-raw '{
        "method": "post",
        "url": "https://yourdomain.com/api/hn",
        "body": {"min": 200, "exclude": "$state.sent"},
        "repeat_every": [1, "hour"]
    }'
    ```

    ```bash CLI theme={null}
    npx @booper/cli run \
      POST https://yourdomain.com/api/hn \
      --data 'min=200' \
      --data 'exclude=$state.sent' \
      --every '1 hour'
    ```
  </CodeGroup>
</Accordion>