Unicorn History Server (UHS)

Disclaimer: This project is currently in development and is in the pre-alpha phase. We warmly welcome all input, contributions, and suggestions.

Unicorn History Server (UHS) is an ancillary service for K8S Clusters using the Yunikorn Scheduler to persist the state of a Yunikorn-managed cluster to a database, allowing for long-term access to historical data of the cluster's operations (e.g. to view past Applications, resource usage, etc.).

Installation

Note: This project is built on G-Research's forks of the Apache YuniKorn project. Please refer to the following G-Research repositories for the relevant forks: yunikorn-core, yunikorn-k8shim, and yunikorn-scheduler-interface.

Quickstart

Use the following make commands to run UHS for a quick test. These commands will install all the necessary dependencies in kind.

Prerequisites

Make sure you have the following dependencies installed:

• Docker - containerization platform.
• kind - tool for running local Kubernetes clusters using Docker container "nodes".
• Go v1.22+ programming language used to build the Unicorn History Server.

Setup local cluster and install dependencies:

start all dependencies - if you are using kind as your K8S cluster manager:

make kind-all

If you want to use minikube for your cluster:

env CLUSTER_MGR=minikube make minikube-all

start the Unicorn History Server:

make run

Architecture

The Unicorn History Server (UHS) is a standalone service that enhances the capabilities of the Yunikorn Scheduler by providing long-term persistence of cluster operational data. It achieves this by listening for events from the Yunikorn Scheduler and persisting them to a database.

UHS is composed of three main components:

1. Event Collector: This component is responsible for listening to events stream API from the Yunikorn Scheduler
   and persisting them to the database. It ensures that all significant operations performed by the scheduler are recorded for future analysis.

2. REST API: This component serves as the interface for retrieving historical data. It provides endpoints that return data about past applications, resource usage, and other operational metrics. Also provides querying capabilities to filter and retrieve specific data.

3. Web Frontend: This component enhances the existing Yunikorn Web interface by providing additional features that utilize the historical data stored by UHS. It is loaded on the application page of Yunikorn Web. The home page served by UHS will display instructions on how to connect UHS to the YuniKorn Web. More details on the web component is available here.

By integrating these components, UHS provides a comprehensive view of the historical operations of a Yunikorn-managed cluster, enabling detailed analysis and insights.

Contributing

We welcome and appreciate your contributions!

General

Report issues, request features, and ask questions using GitHub Issues.

Code Contributions

If you'd like to contribute code, please follow these guidelines:

Open an Issue: Before starting work, ensure there is a relevant issue that your contribution addresses. If none exists, consider creating one to discuss the proposed changes or features.

Fork the Repository: Create your own fork of the repository to make your changes.

Make Your Changes: Implement your changes in your fork, adhering to the project's coding standards and best practices.

Run Tests: Ensure that all tests pass and that your code does not introduce any new issues. All CI checks must pass before submitting your code. Refer to the Testing section for more information.

Submit a Pull Request: Open a pull request to the main repository. Please reference the issue your PR addresses in the description. Clearly explain the changes you’ve made and why they are necessary.

Review Process: Your pull request will be reviewed by the maintainers. Be prepared to make additional changes based on feedback. We usually require at least two maintainers to approve the PR before merging.

Thank you for contributing to our project! Your efforts help us improve and grow the project together ❤️

Testing

Please test contributions thoroughly before requesting reviews. Add and change appropriate unit and integration tests to ensure your changes are covered by automated tests and appear to be correct.

Check Linting

make lint

Run tests

Run tests using kind for cluster manager:

make tests

Run tests using minikube for cluster manager:

env CLUSTER_MGR=minikube make tests

should all succeed without error.

Security

Please see our security policy for details on reporting security vulnerabilities.

License

See the LICENSE file for licensing information.