feat: News aggregator script

[madjin]

ElizaOS News Aggregator Bounty

Project Overview

Create an AI-powered news aggregation system that collects, summarizes, and distributes relevant information about elizaOS development across multiple channels. The system should reduce information silos and improve communication flow within the community.

Objectives

• Collect data from multiple sources (GitHub, Discord, Twitter, on-chain)
• Process and summarize information using LLMs
• Generate digestible daily summaries / show script / weekly newsletter
• Automate the entire pipeline

Data Sources & Requirements

1. On-Chain Data

• Track new tributes via SPL monitoring (Account: AM84n1iLdxgVTAyENBcLdjXoyvjentTbu5Q6EpKV1PeG)
• Market analysis (can use Coingecko API or some eliza / codex plugin)
• Integration with elizas.world metrics (it's open source on github)

2. GitHub Data

• Daily summaries from elizaos.github.io
  • Contributors data: daily/weekly/monthly JSON files
  • Activity summaries
• Expand coverage to all elizaOS organization repositories
• Reference: Current implementation at elizaos.github.io/data/

Here's some references:

• https://elizaos.github.io/data/daily/contributors.json
• https://elizaos.github.io/data/daily/summary.json
• https://elizaos.github.io/data/weekly/contributors.json
• https://elizaos.github.io/data/monthly/contributors.json

Example for daily github summary: https://elizaos.github.io/data/daily/summary.md
All this data is gathered on this repo and updated via a cronjob every day/week/month

3. Discord Integration

• Primary focus on #announcements channel
• Selective monitoring of key discussion channels
• (WIP) Build upon existing discord-summarizer implementation
• (WIP) Integration with workgroups tracking (elizaOS/workgroups)

4. Twitter/X Monitoring

Key accounts to track:

• @daosdotfun
• @ai16zdao
• @shawmakesmagic
• Additional community thought leaders (filter for elizaOS / open source AI news though)

Technical Requirements

• Implement automated data collection pipelines
• Use existing tools where possible:
  • discord-summarizer
  • agent-twitter-client or twitter-scraper-finetune
• Store processed data in standardized format
• Implement error handling and monitoring
• Provide API endpoints for data access

Deliverables

1. Data collection scripts for each source
2. LLM-powered summarization pipeline
3. API for accessing processed data
4. Documentation for maintenance and expansion
5. Deployment instructions

Success Metrics

• Daily summaries generated without manual intervention
• Coverage of all specified data sources
• Accuracy of summarizations
• System uptime and reliability
• Community engagement with generated content

Resources

• elizaOS GitHub organization
• Community Discord
• Technical documentation
• elizaos.github.io/eliza/community/Discord/

[madjin]

This process is automated using the ElizaOS GitHub Contributor Analytics Generator which produces the needed daily summaries suitable to feed into the AI writers' room for episode generation.

The content is aggregated daily at 5PM Eastern from the GitHub contributions. It includes authors, topics, and technical summaries of the important contributions that are to be mentioned on the next AI Podcast episode.

This helps us make AI shows relevant to our interests with up to date info

[madjin]

bootoshi working on a script that'd be useful for this, using twitter lists

can hack something together with this

• https://github.com/elizaOS/agent-twitter-client
• https://github.com/ai16z/twitter-scraper-finetune

[madjin]

odi made a discord bot that can sync discord channels for processing: https://github.com/ai16z/discord-summarizer

it's not fully automated yet

[odilitime]

Just drafted a node_twitter_sync (nitter/rss) so we sync X number of twitter accounts to the data lake. Will commit up the code tmw.

[bozp-pzob]

Got an automated POC that works with twitter/github. Will get discord/onchain up tomorrow.

[odilitime]

good because not sure where I was going to find an easy way to get onchain data. Going to talk to TimShel about the Elizas.world data next

[madjin]

Got an automated POC that works with twitter/github. Will get discord/onchain up tomorrow.

Me or @odilitime can help you through discord data! Can probably speed things up. It just needs a bit more automation

[bozp-pzob]

I chose to do what I have done in node. I can reference what has been done with the python discord library though which will help speed up development