Log Capture & Inspection

Scalabe and Multi-threaded Go Backend for ingesting bulk logs, logs from different services, and also via a direct HTTP log publisher endpoint, all with advanced analytics filtering and a clean UI Interface.

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation & usage

About The Project

This solution provides a Scalable and Efficient implementation for log ingestion and analysis, leveraging the Light-weight multithreading power of Golang and search-indexing capability of ElasticSearch for effective log ingestion, storage, and analysis. It is capable of accepting logs from two distinct sources: a Kafka queue(that moves down to taking logs from various different Services) or direct HTTP API requests. It also offers a web-based user interface for log analysis, supporting various filters and search queries including regex. Architecture Diagram
Project Video

Features

• Kafka queue for streamlined Log processing
• Regular expression search on all fields
• Text-search across all fields
• Efficient search queries leveraging Elastic DB
• Scalable & Efficient Storage using sharding provided by Elastic DB
• Search in given time range
• HTTP endpoint for posting logs
• Ingestion Buffer & Batch processing
• Filters on specific fields
• Combination of all filters & search

Built With

• Golang
• Python
• ElasticSearch
• Kafka
• Gin
• React

Getting Started

Prerequisites

• Python
• Node JS & npm
• Golang
• Docker (Other services can be installed directly or with Docker)

Installation & usage

• Clone the repo

• Start the dependencies, you can use the docker-compose to setup:

  docker compose up -d

• You can check if all the services (ElasticSearch, Zookeeper, Kafka) are up and running by

  docker ps

• Setup Log ingestion server

  1. Go to log-server directory

       cd log-server/

  2. Install golang dependencies

     go mod download

  3. Install Python dependencies

     pip install -r requirements.txt

  4. Start the ingestion server

     cd server
     GIN_MODE=release go run .

     The server should now be up and running on http://localhost:3000.

  5. This script is like a stress test, for a simultaneous bulk log insertion, the current value is 500, but can be modified as per requirements. Configure LOGS_LENGTH in log-server/tests/performance_test.py.

     python tests/performance_test.py

• Setup UI

  1. Go to frontend directory

       cd frontend/

  2. Install NPM dependencies

     npm install

  3. Start the React app

     npm start

  4. Frontend should be up at - http://localhost:3006

• Simulate Log Publishers (logs via http endpoint)

  1. Assuming all the services are up and running.

  2. Start publisher script. Go to log-producers

     cd log-server/log-producers

     Start a producer to simulate service using --topic option in different shells. Configured topics: auth,database,email,payment,server,services, for example

     python producer.py --topic payment
     python producer.py --topic auth