[0011] Internal Development Platform for CI

0011-idp-4-ci.md

@@ -0,0 +1,69 @@
+# CI Pipeline Management through Custom Definitions
+
+## Summary
+
+This RFC proposes the development of a Custom Definitions Management Engine (CDME) to boost the efficiency, reliability, and security of our Continuous Integration (CI) pipelines. By providing an extensible API for CI job orchestration, CDME's aims to: 
+1. streamline workflow deployment across diverse infrastructure, making the process transparent and straightforward for developers; and to
+2. empower developers to craft CI workflows in compliance with a defined policy. 
+
+It will initially target Buildkite and GitHub Actions APIs, primarily seeking to unify under a single infrastructure. It may later be extender to support other APIs.
+
+## Motivation
+<!-- What are the hypotheses behind how this change meets the needs of a PRD? -->
+
+Developers tend to favor secure, reliable and efficient CI/CD, qualities that help accelerate time to production by allowing iterative development processes on realistic infrastructure underlay.
+
+A point requiring attention relates to the efficient handling of workflow orchestration. That is, currently there are two different CI infrastructure providers, **Buildkite** and **GitHub Actions**; the former supporting Mina daemon artifacts, the latter O1js. ***Managing particular requirements in heterogeneous CI environments delays development, obfuscates tests and hinders optimization of overall CI process***.
+
+***It is clear that in order to deliver efficiency, reliability and security to an unified CI an overseeing entity must be introduced***.
+
+<!-- What is the expected outcome or outcomes and what are the exit conditions along the way?  -->
+Such is represented by a Custom Definitions Management Engine (CDME), which will provide high-level abstractions for the efficient development of CI pipelines. Such abstractions are offered as **Templates** to developers, and seek to empower them to craft and test their own CI workflows by abstracting away details of the CI infrastructure.
+
+<!-- Include assessment criteria and ideally measurements that will either verify this hypothesis or yield opportunities to halt and abandon the implementation. -->
+A successful implementation of a CDME may me measured by:
+- Supporting existing CI pipelines via the new abstractions.
+- Reducing the boilerplate of Pipeline/Test definitions by 30%.
+
+<!-- What are we optimizing for in this particular solution? Reflect on alternatives in a later section. -->
+Ultimately, CDME will be the first implementation of the new approach towards CI, which consists on delegating to the Platform Engineering Team the development of tools, workflows and solutions that empower developers to craft amazing software.
+
+<!-- What use cases does it support? -->
+
+## Detailed Design
+The following compose the proposed CDME:
+
+1. A **Pipeline Custom Structure** describing the desired-state of a Pipeline following a predefined schema. Such file details execution steps and required files in an explicit and easy to read format without dynamic content. Its structure is governed by schemas from the in-house definitions to use.
+2. **In-house API Definitions**. A collection of efficient, reliable and secure templates backed by API schemas from Target APIs (i.e., Buildkite and GitHub Actions). These libraries are used as templates by the Pipeline Custom Structure in order to describe workflows.
+3. **Exporter Engine**. Effectively renders the generated Custom Definition into a call to a specific Target API.
+4. **Target API**. Buildkite and GitHub Actions.
+
+
+
+## Test Plan and Functional Requirements
+
+1. **Testing Goals**: Ensure the CDME meets all functional requirements for diverse CI pipeline management, focusing on security, reliability, and user experience.
+2. **Testing Approach**: Utilize a combination of unit, integration, and performance tests, employing automated testing tools where applicable.
+3. **Scope**: Cover all aspects of the CDME, from individual module functionality to the overall integration with existing CI services.
+4. **Requirements**: Adhere to relevant compliance standards, conduct thorough security testing, and simulate real-world user scenarios.
+5. **Resources**: Allocate necessary environments, data sets, and tools for comprehensive testing.
+
+## Drawbacks
+
+Implementing the CDME might initially require significant resources and could potentially disrupt existing CI workflows during the transition phase.
+
+## Rationale and Alternatives
+
+- The CDME offers a unified, scalable solution, addressing the current fragmentation in CI pipeline management.
+- Alternative methods, like continued use of Dhall or other third-party tools, were considered less efficient and secure.
+- Not implementing this change could hinder productivity and lead to security vulnerabilities in CI processes.
+
+## Prior Art
+
+<!-- Investigate similar initiatives in other blockchain projects or CI management tools, evaluating their approaches to CI pipeline optimization. This exploration will help in understanding the broader context and potential pitfalls or successes of similar endeavors. -->
+
+## Unresolved Questions
+
+- Finalize the specific technical aspects of the CDME's architecture and API interfaces during the RFC process.
+- Determine the full scope of CI services and pipeline configurations to support, potentially expanding post-RFC.
+- Address related issues such as long-term maintenance and support, considering future expansions and technology shifts.