Merge pull request #62 from Ayush272002/feat/docs

add documentation for the project

README.md

@@ -1,91 +1,134 @@
-# Turborepo Template
+# Automarker
 
-This is a Turborepo template.
+## How it Works
+As the name suggests its an automarking application which allows the teacher to publish an assignment by uploading a skeleton code, marking script and dockerfile which will have the steps how to containerise the submission and run the marking script which would be uploaded to upload care (AWS S3 could be also used to store them). Once the student of that particular course logs in they would be able to download the skeleton code and fill in their logic for solving the assigned problem, when submitted, the assignment would be uploded to upload care sdk and a link would be provided which is then sent to the backend. The backend sends the submssion details as a payload to a kafka queue which is then picked by the engine at its own pace and then the engine SSH into an AWS EC2 instance and runs the docker container and the marking scipt over there and gathers the log and filters the marks as the marks and then updates it into the database.
 
-## What's inside?
+Currently the engine is configured to 4 tests and the log should be in the format `Number of tests passed: 3` which can be configured and made more dynamic.
 
-This Turborepo includes the following packages/apps:
+NOTE : Docker and dos2unix must be installed in the EC2 instance. 
+- Docker installation guide is [here](https://docs.docker.com/engine/install/)
+- for installing dos2unix use the following command
+```shell
+sudo apt-get update && sudo apt-get install -y dos2unix
+```
+
+## System Design
+Below is the system design for the application.
+
+![System Design](/images/sysDesign.png)
+
+
+
+## Tech Stack
 
-### Apps and Packages
+#### Frontend
+- **Framework**: Next.js  
 
-- `docs`: a [Next.js](https://nextjs.org/) app
-- `web`: another [Next.js](https://nextjs.org/) app
-- `storybook`: [React.js](https://react.dev/) app with storybbok initialised, it can be used as a react app but more specifically wanted it to test components independently using storybook
-- `backend`: [Express.js](https://expressjs.com/) app and [esbuild](https://esbuild.github.io/) as the bundler
-- `@repo/db and @repo/db_serverless`: both are initialised with [prisma](https://www.prisma.io/) the serverless can be used with hono or any other serverless function, and both of them exposes the prisma client.
-- `@repo/ui`: a stub React component library shared by both `web` and `docs` applications which includes [shadcn](https://ui.shadcn.com/) and [MUI - material ui](https://mui.com/material-ui/getting-started/) and [tsup](https://tsup.egoist.dev/) as the bundler
-- `@repo/eslint-config`: `eslint` configurations (includes `eslint-config-next` and `eslint-config-prettier`)
-- `@repo/typescript-config`: `tsconfig.json`s used throughout the monorepo
-- `@repo/tailwind-config`: `tailwind.config.js`s used throughout the monorepo
+#### Backend
+- **Framework**: Node.js with Express.js  
+- **Microservices**: Node.js  
 
-Each package/app is 100% [TypeScript](https://www.typescriptlang.org/).
+#### Communication
+- **Asynchronous Messaging**: Apache Kafka  
 
-### Utilities
+#### Database
+- **Database**: PostgreSQL  
+- **ORM**: Prisma  
 
-This Turborepo has some additional tools already setup for you:
+#### Development Workflow & Code Quality
+- **Monorepo Management**: Turborepo  
+- **Pre-commit Hooks**: Husky  
+- **Code Formatting**: Prettier  
+- **Linting**: ESLint  
 
-- [TypeScript](https://www.typescriptlang.org/) for static type checking
-- [ESLint](https://eslint.org/) for code linting
-- [Prettier](https://prettier.io) for code formatting
-- [Shadcn](https://ui.shadcn.com/) for ui design and components
-- [MUI - material ui](https://mui.com/material-ui/getting-started/) for ui design and components
-- [Husky](https://typicode.github.io/husky/) for pre-commit hook
-- [Lint-Staged](https://www.npmjs.com/package/lint-staged) to run linters against staged git files
+#### DevOps & Infrastructure
+- **CI/CD**: GitHub Actions  
+- **Containerization**: Docker  
+- **Cloud**: AWS EC2  
 
-### Clone
+#### Monitoring & Performance
+- **Metrics Collection**: Prometheus  
+- **Visualization**: Grafana  
 
-To clone the repo run
+
+## Running the project locally
+
+Clone the repository and install dependencies.
 
 ```shell
-git clone https://github.com/Ayush272002/Turborepo-Template.git
-cd template
+https://github.com/Ayush272002/Automarker.git
+cd Automarker
 npm i
 ```
 
-### Build
-
-To build all apps and packages, run the following command:
+### Running the Frontend 
 
-```shell
-cd template
-npm run build
+Go to apps/frontend create a `.env` file and fill all the details which are there in the .env.example. A free API Key for upload care could be obtained from [here](https://uploadcare.com/) 
+```
+npm run build:ui
+npm run dev:frontend
 ```
 
-### Develop
+### Running the backend, engine, Prometheus, and Grafana
+
+For this section, you need to have [Docker](https://docs.docker.com/engine/install/) installed. You also need to fill in the required details in the `docker-compose.yml` file, such as the `DATABASE_URL`, `KAFKA_URI`, `KAFKA_USERNAME`, `KAFKA_PASSWORD`, and AWS credentials.
 
-To develop all apps and packages, run the following command:
+- A free Apache Kafka instance can be obtained from [Aiven Kafka](https://aiven.io/).
+- For AWS keys, you need to create an [AWS EC2 instance](https://aws.amazon.com/), and obtain an SSH private key for SSH access to EC2.
+
+Once all the necessary details are filled in, go to the root directory of the project and run:
 
 ```shell
-cd template
-npm run dev
+docker-compose up
 ```
 
-### Remote Caching
+This will launch the following services:
 
-Turborepo can use a technique known as [Remote Caching](https://turbo.build/repo/docs/core-concepts/remote-caching) to share cache artifacts across machines, enabling you to share build caches with your team and CI/CD pipelines.
+- **Backend**: The backend service will run the API that handles incoming requests and communicates with other services like Kafka and the database.
+- **Engine**: The engine service processes the submissions, runs the Docker container, executes the marking script, and updates the marks.
+- **Prometheus**: A monitoring service that collects metrics and stores time-series data about the health of the application.
+- **Grafana**: A visualization service that displays data collected by Prometheus, which can be used for monitoring and performance tracking.
 
-By default, Turborepo will cache locally. To enable Remote Caching you will need an account with Vercel. If you don't have an account you can [create one](https://vercel.com/signup), then enter the following commands:
+### Accessing the Services
 
-```shell
-cd template
-npx turbo login
-```
+- **Backend API**: Once the containers are running, you can access the backend API at `http://localhost:8000` (configured in the `docker-compose.yml` file).
+- **Prometheus**: Access the Prometheus web UI at `http://localhost:9090` to view the collected metrics.
+- **Grafana**: Access Grafana at `http://localhost:3001` with the default admin credentials (`admin`/`admin`). You can configure it to connect to Prometheus for monitoring data.
+
+---
 
-This will authenticate the Turborepo CLI with your [Vercel account](https://vercel.com/docs/concepts/personal-accounts/overview).
+### Stopping the Services
 
-Next, you can link your Turborepo to your Remote Cache by running the following command from the root of your Turborepo:
+To stop all the services running in Docker containers, use the following command:
 
 ```shell
-npx turbo link
+docker-compose down
 ```
 
-## Useful Links
+This will shut down the running services and remove the containers.
+
+---
+
+### Troubleshooting
+
+- **Missing Docker Compose**: Ensure that Docker Compose is installed. You can verify by running `docker-compose --version`. If not installed, follow the installation instructions [here](https://docs.docker.com/compose/install/).
+
+- **Environment Variables**: Make sure all environment variables in the `.env` file (for the frontend and backend) are properly filled in with the correct values, such as Kafka URI, AWS credentials, database connection strings, etc.
+
+- **Docker Build Issues**: If you encounter issues while building or running Docker containers, check the logs for any errors. You can view the logs using:
+
+  ```shell
+  docker-compose logs
+  ```
+
+---
+
+### Additional Configuration
+
+If you need to scale the services (e.g., running multiple instances of the backend or engine), you can modify the `docker-compose.yml` file accordingly, increasing the `replicas` count for each service.
+
+For Prometheus and Grafana configurations, refer to their respective official documentation for advanced setup, such as adding more data sources or setting up custom dashboards in Grafana.
 
-Learn more about the power of Turborepo:
+## Contributing
+To contribute to the repository, fork the repository, make a new branch (named something relevant to the feature you are adding / bug you are trying to fix), commit any changes you want to commit to your repository, and then send in a pull request to this repository.
 
-- [Tasks](https://turbo.build/repo/docs/core-concepts/monorepos/running-tasks)
-- [Caching](https://turbo.build/repo/docs/core-concepts/caching)
-- [Remote Caching](https://turbo.build/repo/docs/core-concepts/remote-caching)
-- [Filtering](https://turbo.build/repo/docs/core-concepts/monorepos/filtering)
-- [Configuration Options](https://turbo.build/repo/docs/reference/configuration)
-- [CLI Usage](https://turbo.build/repo/docs/reference/command-line-reference)

package.json

@@ -12,6 +12,7 @@
     "db:migrate": "cd packages/db && npx prisma migrate dev && cd ../..",
     "build:frontend": "cd apps/frontend && npm run build && cd ../..",
     "start:frontend": "cd apps/frontend && npm run start && cd ../..",
+    "dev:frontend": "cd apps/frontend && npm run dev && cd ../..",
     "build:backend": "cd apps/backend && npm run build && cd ../..",
     "start:backend": "cd apps/backend && npm run start && cd ../..",
     "build:engine": "cd apps/engine && npm run build && cd ../..",