Skip to content

Latest commit

 

History

History
83 lines (56 loc) · 3.16 KB

README.md

File metadata and controls

83 lines (56 loc) · 3.16 KB
Raw

SolAgg - a Solana Blockchain Data Aggregator

A mini Data aggregator software that collects and processes data from the Solana blockchain. The goal is to create a system capable of retrieving transaction and account data for the ongoing epoch.

SolAgg Preview

or watch the video

Features

  1. Data Retrieval: capable of retrieving transaction and account data from the Solana blockchain on devnet or testnet. Utilise Solana's API or SDK to interact with the blockchain and fetch relevant data.

  2. Data Processing: process the retrieved data efficiently. This includes parsing transaction records, extracting relevant information such as sender, receiver, amount, timestamp, etc., and organising data into a structured format for further analysis and queries.

  3. Data History: Aggregating data from the current epoch and onwards. Exclude historical data to focus on recent transactions and account changes. Ensure the data aggregator provides real-time updates by continuously monitoring the blockchain for new transactions and account changes.

  4. Data Storage: Current in-memory structure that offers scalability, reliability, and fast query capabilities.

  5. API Integration: RESTful API layer to expose the aggregated data to external systems and applications. The API support various queries to retrieve transaction history, account details, and other relevant information.

Architecture and Advantages

  • Work stealing based multithreaded

    • Multithreaded data ingestion from mainnet and/or devnet transactions in realtime, and verified for rate-limited Solana RPC nodes, ~15x performance boost for common single threading data collectors.

Notes on Design and Implementation

  1. using solana RPC client
  2. using Rust typed objects and serde
  3. using solana RPC websocket for continuously monitoring
  4. using Arrow for in-memory storage and SQL query
  5. using tokio and type-safe wrapper for unified API layer

Usage of API endpoints

  • query the current number of total transactions in curruent epoch
curl -sS http://127.0.0.1:3666/transactions/count
  • query the transaction by id
curl -sS 'http://127.0.0.1:3666/transactions?id=fGLvYwnzu8wNbzKmFBJuwNZhcVXuoh4ynpcQEBsRoKX14CoYDtAZd9SCYayaR63X36Sv2sTiXW8yvhmYgH8Ux7A'
  • query the transactions by day
curl -sS 'http://127.0.0.1:3666/transactions?day=16/07/2024'
  • Arbitrary SQL query
curl -X POST -d 'select count(1) as count from transactions where fee=5000' -sS http://127.0.0.1:3666/sql
curl -X POST -d 'select block_time,fee from transactions where fee>5000 limit 2' -sS http://127.0.0.1:3666/sql

Build and Run

(NOTE: assumed you have installed the Rust nightly toolchain in your dev env.)

  1. start the solagg in release mode in the repo root for the perf eval
cargo run --release -- start
  1. start the solagg with debug logging
RUST_LOG=solagg=debug cargo run --release -- start
  1. start the solagg with trace logging(WARN: too many stdou)
RUST_LOG=solagg=trace cargo run --release -- start

Test

Benchmark

Outlook