Word Explorer

Main dependencies:

• HTTP Server: fiber
• Database : elasticsearch
• Message Broker: kafka
• Cron: gocron

Project Layout

.
├── README.md
├── consumer
│   ├── Dockerfile
│   ├── cmd
│   │   └── main.go
│   ├── go.mod
│   ├── go.sum
│   ├── internal
│   │   └── vocabulary
│   │       ├── consumer.go
│   │       ├── model.go
│   │       └── service.go
│   └── pkg
│       └── kafka_reader.go
├── docker-compose.yml
├── ingester
│   ├── Dockerfile
│   ├── cmd
│   │   └── main.go
│   ├── credentials.json
│   ├── go.mod
│   ├── go.sum
│   ├── hash.txt
│   ├── internal
│   │   └── vocabulary
│   │       ├── google-sheet-client.go
│   │       ├── model.go
│   │       ├── publisher.go
│   │       └── service.go
│   ├── pkg
│   │   ├── cron_client.go
│   │   ├── hash.go
│   │   ├── kafka_writer.go
│   │   └── string.go
│   └── token.json
└── word-api
    ├── Dockerfile
    ├── cmd
    │   └── api
    │       └── main.go
    ├── go.mod
    ├── go.sum
    ├── internal
    │   └── vocabulary
    │       ├── handler.go
    │       ├── model.go
    │       ├── repository.go
    │       ├── request.go
    │       ├── response.go
    │       └── service.go
    ├── pkg
    │   └── elastic
    │       └── elastic.go
    └── version1-2023-06-01-mapping.json


17 directories, 37 files

Project Architecture

Motivation

Imagine you have English words with their meanings and example sentences in your Excel sheet in your drive. However, because you are really hardworking, these sheets' size is expanding day by day. You can not search words quickly because the sheet is frozen. As a result, you need an application to search your words quickly.

Our aim is to transform data from an Excel sheet to an elastic search to do a fast search. Therefore, let's know our applications:

Ingester: Retrieve data from the Excel sheet and compare them. If there is any change, it sends the changed data as a message to the Kafka broker.

Consumer: Read messages from Kafka Broker and sends these messages to Word API

Word API: Enables us to communicate Elastic Search, which provides full-text search very fast and stores our data.

All these applications are dockerized, and the docker-composed file is used to run Elastic Search, Kafka, Kibana, and Zookeeper containers.