hackernews-notify

HackerNews Word Match Notifications

Hackernews Notify is a dataflow that watches HackerNews articles against a database of words and the email of the interested parties. When a match is found, the dataflow generates an event with the word, article, and email address of the interested party.

Step-by-step

Take a look at the dataflow.yaml to get an idea of what we're doing.

Make sure to Install SDF and start a Fluvio cluster.

1. Run the Dataflow

Use sdf command line tool to run the dataflow:

Run the dataflow:

sdf run --ui

Use --ui to open the Studio.

2. Start the connector:

In a new terminal change direcotry to ./connectors, download the connector binary, and deploy it:

cd ./connectors
cdk hub download infinyon/[email protected]
cdk deploy start --ipkg infinyon-http-source-0.4.3.ipkg --config ./hackernews.yaml

To see the events, run fluvio consume hackernews -B. The connector reads the RSS every 60 seconds.

For additional context, checkout connectors.

3. Test the Dataflow

Produce Requests to user-request topic

Let's add some generic words that we hope to find in the HackerNews article title.

Produce the following entries in user-request topic:

echo '{"word": "from", "email":"[email protected]"}' | fluvio produce user-request
echo '{"word": "from", "email":"[email protected]"}' | fluvio produce user-request
echo '{"word": "the", "email":"[email protected]"}' | fluvio produce user-request
echo '{"word": "AI", "email":"[email protected]"}' | fluvio produce user-request

Consume from Sink Topic

If any of the words match, you should see records in the notify-event topic:

fluvio consume notify-event -B -O json

In this example from matched:

{
  "email": "[email protected]",
  "link": "https://arxiv.org/abs/2407.20292",
  "title": From pixels to planning: scale-free active inference",
  "word": "from"
}
{
  "email": "[email protected]",
  "link": "https://arxiv.org/abs/2407.20292",
  "title": "From pixels to planning: scale-free active inference",
  "word": "from"
}

Check the state

To watch how the window is gradually populated:

show state

See the notify-table state to see the words and associated emails:

show state process-user-request/notify-table/state

Congratulations! You've successfully built and run a dataflow!

Clean-up

Exit sdf terminal and clean-up. The --force flag removes the topics:

sdf clean --force

Stop the connector:

cdk deploy shutdown --name hackernews