diff --git a/0011-idp-4-ci.md b/0011-idp-4-ci.md new file mode 100644 index 0000000..c9367e7 --- /dev/null +++ b/0011-idp-4-ci.md @@ -0,0 +1,69 @@ +# CI Pipeline Management through Custom Definitions + +## Summary + +This RFC proposes the development of a Custom Definitions Management Engine (CDME) to boost the efficiency, reliability, and security of our Continuous Integration (CI) pipelines. By providing an extensible API for CI job orchestration, CDME's aims to: +1. streamline workflow deployment across diverse infrastructure, making the process transparent and straightforward for developers; and to +2. empower developers to craft CI workflows in compliance with a defined policy. + +It will initially target Buildkite and GitHub Actions APIs, primarily seeking to unify under a single infrastructure. It may later be extender to support other APIs. + +## Motivation + + +Developers tend to favor secure, reliable and efficient CI/CD, qualities that help accelerate time to production by allowing iterative development processes on realistic infrastructure underlay. + +A point requiring attention relates to the efficient handling of workflow orchestration. That is, currently there are two different CI infrastructure providers, **Buildkite** and **GitHub Actions**; the former supporting Mina daemon artifacts, the latter O1js. ***Managing particular requirements in heterogeneous CI environments delays development, obfuscates tests and hinders optimization of overall CI process***. + +***It is clear that in order to deliver efficiency, reliability and security to an unified CI an overseeing entity must be introduced***. + + +Such is represented by a Custom Definitions Management Engine (CDME), which will provide high-level abstractions for the efficient development of CI pipelines. Such abstractions are offered as **Templates** to developers, and seek to empower them to craft and test their own CI workflows by abstracting away details of the CI infrastructure. + + +A successful implementation of a CDME may me measured by: +- Supporting existing CI pipelines via the new abstractions. +- Reducing the boilerplate of Pipeline/Test definitions by 30%. + + +Ultimately, CDME will be the first implementation of the new approach towards CI, which consists on delegating to the Platform Engineering Team the development of tools, workflows and solutions that empower developers to craft amazing software. + + + +## Detailed Design +The following compose the proposed CDME: + +1. A **Pipeline Custom Structure** describing the desired-state of a Pipeline following a predefined schema. Such file details execution steps and required files in an explicit and easy to read format without dynamic content. Its structure is governed by schemas from the in-house definitions to use. +2. **In-house API Definitions**. A collection of efficient, reliable and secure templates backed by API schemas from Target APIs (i.e., Buildkite and GitHub Actions). These libraries are used as templates by the Pipeline Custom Structure in order to describe workflows. +3. **Exporter Engine**. Effectively renders the generated Custom Definition into a call to a specific Target API. +4. **Target API**. Buildkite and GitHub Actions. + + + +## Test Plan and Functional Requirements + +1. **Testing Goals**: Ensure the CDME meets all functional requirements for diverse CI pipeline management, focusing on security, reliability, and user experience. +2. **Testing Approach**: Utilize a combination of unit, integration, and performance tests, employing automated testing tools where applicable. +3. **Scope**: Cover all aspects of the CDME, from individual module functionality to the overall integration with existing CI services. +4. **Requirements**: Adhere to relevant compliance standards, conduct thorough security testing, and simulate real-world user scenarios. +5. **Resources**: Allocate necessary environments, data sets, and tools for comprehensive testing. + +## Drawbacks + +Implementing the CDME might initially require significant resources and could potentially disrupt existing CI workflows during the transition phase. + +## Rationale and Alternatives + +- The CDME offers a unified, scalable solution, addressing the current fragmentation in CI pipeline management. +- Alternative methods, like continued use of Dhall or other third-party tools, were considered less efficient and secure. +- Not implementing this change could hinder productivity and lead to security vulnerabilities in CI processes. + +## Prior Art + + + +## Unresolved Questions + +- Finalize the specific technical aspects of the CDME's architecture and API interfaces during the RFC process. +- Determine the full scope of CI services and pipeline configurations to support, potentially expanding post-RFC. +- Address related issues such as long-term maintenance and support, considering future expansions and technology shifts. \ No newline at end of file