Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for multiple projects to be run on the same host #182

Open
chiruzzimarco opened this issue Jun 23, 2021 · 1 comment
Open

Allow for multiple projects to be run on the same host #182

chiruzzimarco opened this issue Jun 23, 2021 · 1 comment
Labels
bug Something isn't working enhancement New feature or request
Milestone

Comments

@chiruzzimarco
Copy link
Contributor

chiruzzimarco commented Jun 23, 2021

Currently derex have serious issues in running multiple projects (or multiple environments of the same project) on the same host since services are on a per-host basis.
The whole reasoning behind the ddc-services command was about having per-host services (a single mysql, rabbitmq, mongodb, elasticsearch serving any project) in order to speed up development time and do not clobber a developer machine working on a lot of different projects. This multitenant behaviour was implemented gradually by:

  • naming mysql and mongodb databases after the project name
  • naming rabbitmq queue after the project name
  • naming docker containers after the project name

This is all nice, except that some issues still persists and others may be unsolvable:

  • the elasticsearch indices should be named after the project name. This involve changing a lot of Openedx Django settings and may even prove impossible without changing some core code
  • since ddc-services is run on a per-host basis it won't account for any particular project. Thus it can't know that a specific project is running Openedx on juniper release and needs mysql 5.6 while a different one is running Openedx on koa release and needs mysql 5.7
  • different environments of the same project cannot run on the same host without changing the project name since they will conflict and overlap. The need to change the project name in order to run the same project twice (production/staging) is conceptually wrong.
  • even if it was in the plan to hash them with the project name, secrets are stille derived from the host main secret only. Thus different projects will share secrets
  • derex mysql and mongodb commands currently act with root privileges in order to be able to create/drop databases and users. This means that a project is authorized to drop another project related database/user. Same applies to rabbitmq queues and MinIO buckets.
  • every project belongs to the same network and use the same internal httpserver. Thus containers from different projects are able to communicate between each other.

For those reasons i suggest removing ddc-services altogether and include all services on a per-project basis. The only case where i see it fit is about host monitoring (still to be implemented).

@chiruzzimarco
Copy link
Contributor Author

chiruzzimarco commented Mar 22, 2022

Elasticsearch indexes names are hardcoded in the edx-platform code here:

https://github.com/openedx/edx-platform/blob/master/cms/djangoapps/contentstore/courseware_index.py#L343
https://github.com/openedx/edx-platform/blob/master/cms/djangoapps/contentstore/courseware_index.py#L449
https://github.com/openedx/edx-platform/blob/master/cms/djangoapps/contentstore/courseware_index.py#L554

If there are other locations, i was not able to find them.
I suggest we replace the courseware_index.py module with a version which tries to take the elasticsearch index name from Django settings and defaults to the original name if the setting cannot be found.

Each supported openedx release needs to have it's own module version customized here.

This will allow us to namespace the index name like we are already doing for the mysql database name for example.

This should be handled in a separate PR so to restrict the scope of this PR. It will be useful regardless of the choices we will take on derex future developments about handling different project environments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant