Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: ADR 15 - repo consolidation #292

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Status
******

Accepted
Superseded by `ADR 14 Project Structure`_

Context
*******
Expand Down Expand Up @@ -104,6 +104,7 @@ Rejected Alternatives
Supercedes `ADR 6 Areas of responsibility`_.

.. _ADR 6 Areas of responsibility: 0006_areas_of_responsibility.html
.. _ADR 14 Project Structure: 0014_project_structure_update.html
.. _event-routing-backends: https://github.com/openedx/event-routing-backends
.. _openedx-aspects: https://github.com/openedx/openedx-aspects
.. _openedx-event-sink-clickhouse: https://github.com/openedx/openedx-event-sink-clickhouse
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Status
******

Draft -> Accepted
Superseded by `ADR 15 Repository Consolidation`_.

Context
*******
Expand Down Expand Up @@ -102,6 +102,7 @@ Supersedes `ADR 8 Project Structure`_.

.. _ADR 6 Areas of responsibility: 0006_areas_of_responsibility.html
.. _ADR 8 Project Structure: 0008_project_structure.html
.. _ADR 15 Repository Consolidation: 0015_repository_consolidation.html
.. _event-routing-backends: https://github.com/openedx/event-routing-backends
.. _openedx-aspects: https://github.com/openedx/openedx-aspects
.. _openedx-event-sink-clickhouse: https://github.com/openedx/openedx-event-sink-clickhouse
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
15. Repository Consolidation
############################

Status
******

Draft -> Accepted

Context
*******

This ADR builds upon, and in some cases, supersedes `ADR 8 Project Structure`_ and `ADR 14 Project Structure Update`_.

The Aspects project has evolved rapidly over the past 2 years, and has been through a number of iterations. The project has been split into multiple repositories, broken up along functional lines and architectural domains. As the project has developed, this has led to a number of issues:

- Placing our custom Python code in the Tutor plugin has made testing challenging and encouraged some bad practices such as placing Jinja variables directly in the code. It also locks us into Tutor as the only supported deployment method.
- Version management across repositories requires that all repositories be kept in sync, which is a manual and error prone process.
- There is a fair amount of toil as we have to manage multiple repositories, each with their own CI/CD pipelines, dependency updates, documentation, and issue tracking. While not terrible, this is a significant amount of overhead for the number of maintainers we have.


Decisions
*********

This ADR seeks to address the above issues by consolidating the Aspects project from 5 repositories to 3 by moving the Python code portions into `openedx-aspects`. By making that an installable project which is then used by `tutor-contrib-aspects` we can simplify testing and release of this code, improve code quality, and simplify using Aspects in non-Tutor environments significantly.

**openedx-aspects**

This repo (`openedx-aspects`_) remains the primary Aspects repository where the project decisions and documentation are stored. However, we will also move the Python code for Superset asset management and authentication/authorization into this repository as well as the contents of the `aspects-dbt` and `xapi-db-load` repositories. Those repositories can then be deprecated and archived.

This repo should will also be used to store any common code or configuration between Tutor and non-Tutor Aspects deployments, such as the Alembic migrations needed to bootstrap the Aspects databases.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This repo should will also be used to store any common code or configuration between Tutor and non-Tutor Aspects deployments, such as the Alembic migrations needed to bootstrap the Aspects databases.
This repo will also be used to store any common code or configuration between Tutor and non-Tutor Aspects deployments, such as the Alembic migrations needed to bootstrap the Aspects databases.


**tutor-contrib-aspects**

`tutor-contrib-aspects`_ will be updated to use the `openedx-aspects`_ repository as a dependency. This will allow us to keep the Tutor plugin as a separate repository, but will allow us to keep the Python code encapsulated in an easier to test environment, force us to move the Tutor Jinja configuration variables into a traditional configuration layer, where they belong. This will also allow us to greatly simplify the Tutor plugin and separate the concerns of code and configuration.

Care must be taken to ensure that the extension mechanisms for Superset assets, localization, and authentication / authorization are not broken by this change.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some (if not most) of those things will be broken, I think we should keep compatibility with custom DBT repositories, but I'm worried that not all extension mechanisms can be kept.


**aspects-dbt**

This repository will be archived and the contents moved into `openedx-aspects`_. We will need to make sure that the mechanism of deploying the dbt models does not break our mechanisms of extending them, since currently the dbt models are extended by creating a new dbt project that has a dependency on the `aspects-dbt` repository itself.

**xapi-db-load**

This repository will be archived and the contents moved into `openedx-aspects`_. Since this repository is now tightly bound to the Alembic migrations that will be moved to `openedx-aspects`_, it makes sense to consolidate them and make this a properly maintained project.

**platform-plugin-aspects**

This repository will remain as it is.

Consequences
************

* The Aspects project will be easier to maintain and develop for, as the code will be consolidated into a fewer interdependent repositories.

* The Aspects project will be easier to test, as the Python code will be encapsulated in a separate repository.

* The Aspects project will be easier to deploy outside of Tutor.

* The Aspects project will be easier to document, as the code and configuration will be more clearly separated.

Rejected Alternatives
*********************

* Leave things as they are.


Supersedes
**********

* Supersedes `ADR 6 Areas of responsibility`_.
* Supersedes `ADR 8 Project Structure`_.
* Supersedes `ADR 14 Project Structure Update`_.

.. _ADR 6 Areas of responsibility: 0006_areas_of_responsibility.html
.. _ADR 8 Project Structure: 0008_project_structure.html
.. _ADR 14 Project Structure Update: 0014_project_structure_update.html
.. _event-routing-backends: https://github.com/openedx/event-routing-backends
.. _openedx-aspects: https://github.com/openedx/openedx-aspects
.. _openedx-unsupported: https://github.com/openedx-unsupported
.. _tutor-contrib-aspects: https://github.com/openedx/tutor-contrib-aspects
.. _platform_plugin_superset: https://github.com/openedx/platform-plugin-superset/