feat: add Google Cloud Storage support (GCS)

[shadinaif]

TL;DR - This is to add support to GCS files backend by adding a new option (gcs) to ORA2_FILEUPLOAD_BACKEND

JIRA: ---

What changed?

• ORA2_FILEUPLOAD_BACKEND environment variable accepts a new value gcs in addition to the existing options (s3, filesystem, swift, and django)
• No changes to the requirements. It is assumed that you already installed django-storages[google] instead of the default django-storages which will include google-cloud-storages

Important to know!
Using GSC with ORA means that edx-platform is already configured to use GCS instead of S3

• It is assumed that you already installed django-storages[google] instead of the default django-storages which will include google-cloud-storages
• It is assumed that DEFAULT_FILE_STORAGE is set to storages.backends.gcloud.GoogleCloudStorage in edx-platform. Other configurations might be needed too
• It is assumed that edx-platform functions directly and seamlessly with GCS without a proxy like MinIO or boto3

Developer Checklist

• Reviewed the release process
• Translations and JS/SASS compiled~~ not needed
• Bumped version number in openassessment/__init__.py and package.json

Testing Instructions

Using tutor development:

• Prepare your GSC storage, make sure to have a dedicated bucket for file upload
• Configure edx-platform to use GCS, including the installation of django-storages[google]
• Configure ORA2 to use GCS by setting ORA2_FILEUPLOAD_BACKEND to gcs
• Configure the dedicated bucket by setting FILE_UPLOAD_STORAGE_BUCKET_NAME
• Test uploading files from ORA2 and previewing them
• Test previewing uploaded files in ora2 MFE
• Verify that files exist in the dedicated storage
• Test deleting files
• Verify that files were deleted from the dedicated storage

Reviewer Checklist

Collectively, these should be completed by reviewers of this PR:

• I've done a visual code review
• I've tested the new functionality

FYI: @openedx/content-aurora

[openedx-webhooks]

Thanks for the pull request, @shadinaif!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/2u-aurora. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[itsjeyd]

@shadinaif Thank you for this contribution! Let us know when the changes are ready for review.

[itsjeyd]

Hey @shadinaif, just checking in to see if you're still planning to continue working on this PR?

[shadinaif]

Thank you for the reminder @itsjeyd . Yes, it's on production and working fine. I'll rebase and set to review soon

[itsjeyd]

@shadinaif Sounds good, thanks for the update.

[itsjeyd]

Hey @shadinaif, do you have any updates about when you'll be able to come back to this PR?

[shadinaif]

Thanks for the follow-up @itsjeyd , I'll get back to it and complete it on Tuesday, June 4, 2024 I promise. I've added it to my schedule. Sorry for the delay 🙏🏼

[itsjeyd]

Sounds good, thanks for the update @shadinaif.

[itsjeyd]

@shadinaif It looks like tests will need some further tweaking here. Could you have a look?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.12%. Comparing base (8b320d6) to head (8fd1c39).
Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2193      +/-   ##
==========================================
+ Coverage   95.09%   95.12%   +0.02%     
==========================================
  Files         195      196       +1     
  Lines       21479    21584     +105     
  Branches     1931     1935       +4     
==========================================
+ Hits        20425    20531     +106     
  Misses        787      787              
+ Partials      267      266       -1     

Flag	Coverage Δ
unittests	95.12% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[itsjeyd]

Hey @shadinaif, thanks for working on the tests.

It looks like some of them are still failing. Will you be taking another look?

[shadinaif]

Thanks for the reminder @itsjeyd , I'll do in a couple of days 👍🏼

[itsjeyd]

@shadinaif Just checking in to see if you're still planning to continue working on this PR? If you don't think you'll get to it in the short term, it might make sense to close it for the time being.

[shadinaif]

this is not related to the PR, it just makes it easier to run tests for a single file or object locally

[shadinaif]

@itsjeyd thank you for your patience on this. I've added the needed tests, and I also had to fix some linter errors not related to the PR. All green now

[itsjeyd]

@shadinaif Great! I'm marking this PR as ready for review now.

[itsjeyd]

Hi @openedx/2u-aurora team, this PR is ready for engineering review.

@feanil I can't formally request a review from the Aurora team, could you please update the team's permissions for this repo to make that possible?

[feanil]

Added the team to the repo and tagged them for review.

[pomegranited]

Hi @shadinaif , I'm having trouble getting Google Cloud credentials configured to test this change. Are you planning to submit a GCS integration PR to the platform?

I found documentation for initializing the GS_CREDENTIALS setting, but since it requires initializing that setting as a google.oauth2.service_account.Credentials object, I can't configure it with yml, and would have to modify the platform's envs/*.py files.

And without platform integration, I don't think we should add it to ORA2 either. Open to discussion though, let me know?

[pomegranited]

This change looks unrelated?

[pomegranited]

This change looks unrelated?

[pomegranited]

nit: This repo is missing the compile-requirements make target, but would you be willing to add it with the diff below? Then you can run make compile-requirements and avoid this unneeded change to the requirements file.

diff --git a/Makefile b/Makefile
index 68cc2a636..aa2ca0bcf 100644
--- a/Makefile
+++ b/Makefile
@@ -43,23 +43,23 @@ $(COMMON_CONSTRAINTS_TXT):
        wget -O "$(@)" https://raw.githubusercontent.com/edx/edx-lint/master/edx_lint/files/common_constraints.txt || touch "$(@)"
        echo "$(COMMON_CONSTRAINTS_TEMP_COMMENT)" | cat - $(@) > temp && mv temp $(@)

-upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade
-upgrade: $(COMMON_CONSTRAINTS_TXT)  ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
+compile-requirements: export CUSTOM_COMPILE_COMMAND=make upgrade
+compile-requirements: $(COMMON_CONSTRAINTS_TXT)  ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in
     # global common_constraints has this pin.
        sed 's/django-simple-history==3.0.0//g' requirements/common_constraints.txt > requirements/common_constraints.tmp
        mv requirements/common_constraints.tmp requirements/common_constraints.txt
        pip install -qr requirements/pip-tools.txt
-       pip-compile --upgrade --allow-unsafe -o requirements/pip.txt requirements/pip.in
-       pip-compile --upgrade -o requirements/pip-tools.txt requirements/pip-tools.in
+       pip-compile ${COMPILE_OPTS} --allow-unsafe -o requirements/pip.txt requirements/pip.in
+       pip-compile ${COMPILE_OPTS} -o requirements/pip-tools.txt requirements/pip-tools.in
        pip install -qr requirements/pip.txt
        pip install -qr requirements/pip-tools.txt
-       pip-compile --upgrade -o requirements/base.txt requirements/base.in
-       pip-compile --upgrade -o requirements/test.txt requirements/test.in
-       pip-compile --upgrade -o requirements/quality.txt requirements/quality.in
-       pip-compile --upgrade -o requirements/test-acceptance.txt requirements/test-acceptance.in
-       pip-compile --upgrade -o requirements/tox.txt requirements/tox.in
-       pip-compile --upgrade -o requirements/ci.txt requirements/ci.in
-       pip-compile --upgrade -o requirements/docs.txt requirements/docs.in
+       pip-compile ${COMPILE_OPTS} -o requirements/base.txt requirements/base.in
+       pip-compile ${COMPILE_OPTS} -o requirements/test.txt requirements/test.in
+       pip-compile ${COMPILE_OPTS} -o requirements/quality.txt requirements/quality.in
+       pip-compile ${COMPILE_OPTS} -o requirements/test-acceptance.txt requirements/test-acceptance.in
+       pip-compile ${COMPILE_OPTS} -o requirements/tox.txt requirements/tox.in
+       pip-compile ${COMPILE_OPTS} -o requirements/ci.txt requirements/ci.in
+       pip-compile ${COMPILE_OPTS} -o requirements/docs.txt requirements/docs.in
        # Delete django pin from test requirements to avoid tox version collision
        sed -i.tmp '/^[d|D]jango==/d' requirements/test.txt
        sed -i.tmp '/^djangorestframework==/d' requirements/test.txt
@@ -70,6 +70,9 @@ upgrade: $(COMMON_CONSTRAINTS_TXT)  ## update the requirements/*.txt files with
        # Delete temporary files
        rm requirements/*.txt.tmp

+upgrade:  ## update the pip requirements files to use the latest releases satisfying our constraints
+       $(MAKE) compile-requirements COMPILE_OPTS="--upgrade"
+

[itsjeyd]

Thanks for picking up the review for this PR @pomegranited!