feat: install atlas in edx_django_service

[OmarIthawi]

Description

Download the openedx-atlas executable for all edx_django_service to be used in make pull_translations

This installs atlas on all Ansible build targets such as Native Installation and Devstack.

Once merged it will make the atlas CLI executable available on all edx_django_service Dockerfiles and native installation such as LMS, Studio, Discovery and others. It will make the following command possible: atlas pull which replaces transifex pull as part of the Translation Infrastructure update OEP-58 project.

Testing

• Re-build a devstack docker image and run atlas pull

Testing instructions:

configuration $ make docker.build.credentials 2>&1 | tee /tmp/cretentials-build-log.txt
docker pull edxops/focal-common:latest
....
 => => exporting layers
 => => writing image sha256:55ad82ed7e10cb0c03aa56d50c45fd5f6fccd7b549fc3baffbf5fa0dfec49c30

# Now, tag the credentials image with the newly built tag
configuration $ docker tag $(grep writing.image /tmp/cretentials-build-log.txt | sed -e 's/ done//' -e 's/^.*writing image sha256://') edxops/credentials:latest

# Go to devstack
configuration $ cd ../devstack
devstack $ make dev.remove-containers
devstack $ make dev.up.credentials
devstack $ docker run -it edxops/credentials:latest atlas --version

# From the credentials container, try atlas
credentials# atlas --version
credentials# rm -rf conf/locale/
credentials# atlas pull translations/credentials/credentials/conf/locale:conf/locale
credentials# ls -R conf/

To reset your local image:

$ docker pull edxops/credentials:latest
$ docker run -it edxops/credentials:latest atlas --version # Should throw an error now

Test results on my machine

build, check and pull logs

Build credentials image:

$ make docker.build.credentials
docker pull edxops/focal-common:latest
latest: Pulling from edxops/focal-common
Digest: sha256:32243e53b16c7a8b333087d75f3503ff1648d3eabd2e142b9cf8e0591e9381fb
Status: Image is up to date for edxops/focal-common:latest
docker.io/edxops/focal-common:latest
docker build -f docker/build/credentials/Dockerfile .
[+] Building 150.0s (12/12) FINISHED
...
 => => writing image sha256:55ad82ed7e10cb0c03aa56d50c45fd5f6fccd7b549fc3baffbf5fa0dfec49c30

Test the help command:

$ docker run -it 55ad82ed7e10cb0 atlas --help
Atlas is a CLI tool that has essentially one command: `atlas pull`

Configuration file:
...

Check the version and pip status:

$ docker run -it 55ad82ed7e10cb0 atlas --version
v0.4.4
$ docker run -it 55ad82ed7e10cb0 pip freeze | grep atlas
openedx-atlas==0.4.4

Test the pull command:

$ docker run -it 55ad82ed7e10cb0 bash -c 'cd /edx/app/credentials/credentials/credentials/conf; rm -rf locale/; atlas pull translations/credentials/credentials/conf/locale:locale; ls -R .'
Pulling translation files
 - directory: translations/credentials/credentials/conf/locale:locale
 - repository: openedx/openedx-translations
 - branch: main
 - filter: Not Specified
Creating a temporary Git repository to pull translations into "./translations_TEMP"...
Done.
Setting git sparse-checkout rules...
Done.
Pulling translation files from the repository...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 4 (delta 2), reused 1 (delta 1), pack-reused 0
Receiving objects: 100% (4/4), 7.47 KiB | 7.47 MiB/s, done.
Resolving deltas: 100% (2/2), done.
Your branch is up to date with 'origin/main'.
Done.
Copying translations from "./translations_TEMP/translations/credentials/credentials/conf/locale" to "./locale"...
Done.
Removing temporary directory...
Done.

Translations pulled successfully!
.:
locale

./locale:
en  fr_CA

./locale/en:
LC_MESSAGES

./locale/en/LC_MESSAGES:
djangojs.po  django.po

./locale/fr_CA:
LC_MESSAGES

./locale/fr_CA/LC_MESSAGES:
djangojs.po  django.po

Configuration Pull Request

Make sure that the following steps are done before merging:

• A SRE team member has approved the PR if it is code shared across multiple services and you don't own all of the services.
• Are you adding any new default values that need to be overridden when this change goes live? [Omar] No, the default value is safe to be used.
  • Update the appropriate internal repo (be sure to update for all our environments)
  • If you are updating a secure value rather than an internal one, file a SRE ticket with details.
  • Add an entry to the CHANGELOG.
• If you are making a complicated change, have you performed the proper testing specified on the Ops Ansible Testing Checklist? Adding a new variable does not require the full list (although testing on a sandbox is a great idea to ensure it links with your downstream code changes).
• Think about how this change will affect Open edX operators. Have you updated the wiki page for the next Open edX release?
  • [Omar] The documentation update is work-in-progress.

References

This contribution is part of the FC-0012 project which is sparked by the Translation Infrastructure update OEP-58.

Up-to-date project overview and details are available in the Approach Memo and Technical Discovery: Translations Infrastructure Implementation document.

Join the conversation on Open edX Slack #translations-project-fc-0012.

Check the links above for full information about the overall project.

Internalization is being rearchitected in Open edX Python, XBlock, Micro-frontend, and other projects. There are a number of immediately visible changes:

• Remove source and language translations from the repositories, hence no .json, .po or .mo files will be committed into the repos.
• Add standardized make extract_translations in all repositories
• Push user-facing messages strings into openedx/openedx-translations.
• Integrate root repositories with openedx/openedx-atlas to pull translations on build/deploy time

Breaking Changes

One of the primary goals of the project is to avoid breaking changes. If you noticed any suspicious code, please raise your concern. But before that, please know the strategy we're following to avoid breaking changes:

• All variable have sane-defaults.
• The changes are backward compatible and don't break any existing deployments.

[openedx-webhooks]

Thanks for the pull request, @OmarIthawi! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[OmarIthawi]

@brian-smith-tcril now this is tested and I think it's worth merging unless we need additional reviewers' 👍🏼

[brian-smith-tcril]

I think we should wait until we implement openedx/openedx-atlas#34 and use pip here

[OmarIthawi]

@brian-smith-tcril I've updated it to install from npm with detailed testing steps.

[brian-smith-tcril]

Have you tested running atlas after running one of the makefile shell commands? If so, could we simplify the testing instructions to use make dev.shell.lms or something instead of using docker run and a sha?

[OmarIthawi]

@brian-smith-tcril I've done the testing and it works with devstack's make dev.shell.credentials.

Please note that LMS isn't included in this build because it doesn't use the edx_django_service role.

LMS openedx-atlas requirement has another strategy:

• feat: install atlas translations tool openedx/edx-platform#32952 (work in progress)

[brian-smith-tcril]

Would it be possible to do this as a part of "install production requirements" or "install extra requirements" instead? It seems that's likely the more common pattern.

[OmarIthawi]

@brian-smith-tcril So we have two options:

• Go to each service like credentials, ecommerce and others and add openedx-atlas to its base.in requirements file, or
• Add the requirement in configuration (this PR)

I realize now that this is a questionable approach to add it here, but I'm open to hear from you and other reviewers.

I'm realizing we should close this pull request and install atlas in individual repos instead.

[brian-smith-tcril]

Go to each service like credentials, ecommerce and others and add openedx-atlas to its base.in requirements file

I think I like this idea best, but I'm very open to any other thoughts/opinions on the matter.

[OmarIthawi]

Thanks! I've became hesitant after completing this PR, so I close it and move on adding atlas to each repo individually.

It'll take more time to add to each individual service, but honestly it's the least surprising place to specify a requirement.

I'll close this PR and start working on the individual repos.

[openedx-webhooks]

@OmarIthawi Even though your pull request wasn’t merged, please take a moment to answer a two question survey so we can improve your experience in the future.