Fix oppia#11465: Migrates the codebase from Python 2 to Python 3 (opp…

…ia#13395)

* Migrates the codebase from Python 2 to Python 3

* fix lint

* fix tests

* fix blog tests

* update pylintrc

* run mypy checks

* run mypy checks

* revert changes to CODEOWNERS

* Test backend error

* Test backend error

* Test backend error

* Test backend error

* Improve coverage

* Improve coverage

* Improve coverage

* fix lint

* fix frontend coverage

* Fix backend tests

* fix mypy checks

* Improve coverage

* Improve coverage

* Fix tests and improve coverage

* fix broken test

* fix lint

* Improve coverage

* Improve coverage

* Improve coverage

* fix learner_progress_services

* Improve coverage and fix types

* Fix mypy

* Fix tests and improve coverage

* Increase coverage

* inline mock_datetime_utcnow

* fix lint

* Split e2e suite to improve performance

* Split e2e suite to improve performance

* Split e2e suite to improve performance

* Split e2e suite to improve performance

* fix datetime tests

* fix tests

* disable incorrect pylint warning

* Improve coverage

* Improve coverage

* Remove elastic search from lighthouse tests

* Fix e2e tests

* simplify metaclass

* Address comments

* try to store on disk

* try to store on disk

* update pylintrc

* Address some comments

* Address some comments

* Address some comments

* Address some comments

* Address more comments

* Fix backend tests

* Fix backend tests

* Address comments

* Address comments

* Merge related edits

* Address more comments.

* Address comments

* Address comments

* Fix backend tests

* Fix backend tests

* Fix backend tests

* Debug portserver

* Address comments

* Address comments

* Address comments

* Fix prereqs

* Add tests

* Address comments

* Fix tests

* Fix tests

* Address comments

* Fix lighthouse tests

* Fix tests and increase coverage

* Fix backend tests

* Fix backend tests

* Small improvements

* Fix lighthouse

* [skip ci] Lighthouse changes

* Fix encode/decode

* Fix backend tests

* Fix ighthouse tests

* Fix lighthouse

* Fix tests

* Fix redis shutdown

* Add debug lines

* Add FE debugging

* Fix e2e and backend tests

* Fix backend tests

* Fix portserver printing

* Fix lighthouse, debug e2e

* Fix portserver hangs

* Fix portserver and add debugging

* Fix debugging

* Fix release

* Debug and fix tests

* Fix backend tests

* Fix frontend

* Fix e2e tests

* Fix FE tests

* Remove debugging statements

* Fix e2e tests

* Fix frontend tests

* Fix tests and address comments

* Fix frontend and e2e tests

* Debug e2e tests

* Fix e2e tests

* Fix e2e and frontend tests

* Change policies

* Remove debugging statements

* Fix e2e errors

* Fix e2e tests

* Reset flake policy for extensions

* Partial mypy

* Fix backend tests

* Fix mypy

* Fix backend tests and linting

* Fix backend tests

Co-authored-by: Vojtěch Jelínek <[email protected]>
Co-authored-by: Vojtěch Jelínek <[email protected]>

.circleci/config.yml

@@ -7,7 +7,7 @@ workflows:
     jobs:
       - setup_and_typescript_tests
 
-var_for_docker_image: &docker_image circleci/python:2.7.17-browsers
+var_for_docker_image: &docker_image circleci/python:3.7.10-browsers
 
 anchor_for_job_defaults: &job_defaults
   working_directory: /home/circleci/oppia

.gcloudignore

@@ -0,0 +1,167 @@
+# Various compiled, temporary, and hidden files
+*~
+*.pyc
+*.pyo
+*.swp
+*.swo
+*.bak
+.*
+# Typescript files in core
+core/**.ts
+# Typescript output log file
+tsc_output_log.txt
+# Python test files
+*_test.py
+# Other folders to ignore
+core/tests/
+node_modules/
+scripts/
+third_party/python3_libs/
+# Some third_party static scripts are directly imported, namely: jquery,
+# jqueryui, angularjs, jqueryui-touch-punch, MathJax, code-mirror,
+# ui-codemirror, d3js, midi-js, ui-map, guppy, skulpt, math-expressions.
+# We exclude some of the scripts that are not directly imported in order to
+# reduce deployed file count.
+# TODO(sll): Find a more structured way of doing this.
+#
+# Please do not remove line below this, as it is used for testing purpose.
+# Third party files:
+third_party/static/angular-audio-1.7.4/
+third_party/static/angular-toastr-1.7.0/
+third_party/static/bootstrap-4.3.1/
+third_party/static/bower-angular-translate-2.18.1/
+third_party/static/bower-angular-translate-interpolation-messageformat-2.18.1/
+third_party/static/bower-angular-translate-loader-partial-2.18.1/
+third_party/static/bower-angular-translate-loader-static-files-2.18.1/
+third_party/static/bower-angular-translate-storage-cookie-2.18.1/
+third_party/static/bower-material-1.1.19/
+third_party/python_libs/google/appengine/
+third_party/python_libs/google/net/
+third_party/python_libs/google/pyglib/
+third_party/python_libs/grpc/
+# CKEditor-4.12.1 plugins in the download from the CKEditor website include
+# only a11yhelp, about, clipboard, colordialog, copyformatting, dialog, div,
+# find, flash, forms, iframe, image, link, liststyle, magicline, pagebreak,
+# pastefromword, preview, scayt, showblocks, smiley, specialchar, table,
+# tableselection, tabletools, templates, widget and wsc. Our code is also using
+# the sharedspace plugin. So, for now, we exclude all others, as well as flash,
+# a11yhelp, about, colordialog, iframe, and anything related to tables, which
+# we definitely don't use.
+third_party/static/ckeditor-4.12.1/plugins/a11yhelp/
+third_party/static/ckeditor-4.12.1/plugins/about/
+third_party/static/ckeditor-4.12.1/plugins/adobeair/
+third_party/static/ckeditor-4.12.1/plugins/ajax/
+third_party/static/ckeditor-4.12.1/plugins/autocomplete/
+third_party/static/ckeditor-4.12.1/plugins/autoembed/
+third_party/static/ckeditor-4.12.1/plugins/autogrow/
+third_party/static/ckeditor-4.12.1/plugins/autolink/
+third_party/static/ckeditor-4.12.1/plugins/balloonpanel/
+third_party/static/ckeditor-4.12.1/plugins/balloontoolbar/
+third_party/static/ckeditor-4.12.1/plugins/bbcode/
+third_party/static/ckeditor-4.12.1/plugins/bidi/
+third_party/static/ckeditor-4.12.1/plugins/cloudservices/
+third_party/static/ckeditor-4.12.1/plugins/codesnippet/
+third_party/static/ckeditor-4.12.1/plugins/codesnippetgeshi/
+third_party/static/ckeditor-4.12.1/plugins/colorbutton/
+third_party/static/ckeditor-4.12.1/plugins/colordialog/
+third_party/static/ckeditor-4.12.1/plugins/devtools/
+third_party/static/ckeditor-4.12.1/plugins/dialogadvtab/
+third_party/static/ckeditor-4.12.1/plugins/divarea/
+third_party/static/ckeditor-4.12.1/plugins/docprops/
+third_party/static/ckeditor-4.12.1/plugins/easyimage/
+third_party/static/ckeditor-4.12.1/plugins/embed/
+third_party/static/ckeditor-4.12.1/plugins/embedbase/
+third_party/static/ckeditor-4.12.1/plugins/embedsemantic/
+third_party/static/ckeditor-4.12.1/plugins/emoji/
+third_party/static/ckeditor-4.12.1/plugins/flash/
+third_party/static/ckeditor-4.12.1/plugins/font/
+third_party/static/ckeditor-4.12.1/plugins/iframe/
+third_party/static/ckeditor-4.12.1/plugins/iframedialog/
+third_party/static/ckeditor-4.12.1/plugins/image2/
+third_party/static/ckeditor-4.12.1/plugins/imagebase/
+third_party/static/ckeditor-4.12.1/plugins/indentblock/
+third_party/static/ckeditor-4.12.1/plugins/justify/
+third_party/static/ckeditor-4.12.1/plugins/language/
+third_party/static/ckeditor-4.12.1/plugins/mathjax/
+third_party/static/ckeditor-4.12.1/plugins/mentions/
+third_party/static/ckeditor-4.12.1/plugins/newpage/
+third_party/static/ckeditor-4.12.1/plugins/panelbutton/
+third_party/static/ckeditor-4.12.1/plugins/placeholder/
+third_party/static/ckeditor-4.12.1/plugins/print/
+third_party/static/ckeditor-4.12.1/plugins/save/
+third_party/static/ckeditor-4.12.1/plugins/selectall/
+third_party/static/ckeditor-4.12.1/plugins/sourcedialog/
+third_party/static/ckeditor-4.12.1/plugins/stylesheetparser/
+third_party/static/ckeditor-4.12.1/plugins/table/
+third_party/static/ckeditor-4.12.1/plugins/tableresize/
+third_party/static/ckeditor-4.12.1/plugins/tabletools/
+third_party/static/ckeditor-4.12.1/plugins/textmatch/
+third_party/static/ckeditor-4.12.1/plugins/textwatcher/
+third_party/static/ckeditor-4.12.1/plugins/uicolor/
+third_party/static/ckeditor-4.12.1/plugins/uploadfile/
+third_party/static/ckeditor-4.12.1/plugins/xml/
+third_party/static/ckeditor-4.12.1/samples/
+third_party/static/ckeditor-4.12.1/skins/kama/
+third_party/static/ckeditor-4.12.1/skins/moono/
+third_party/static/ckeditor-bootstrapck-1.0.0/core/
+third_party/static/ckeditor-bootstrapck-1.0.0/lang/
+third_party/static/ckeditor-bootstrapck-1.0.0/plugins/
+third_party/static/ckeditor-bootstrapck-1.0.0/skins/bootstrapck-dev/
+third_party/static/ckeditor-bootstrapck-1.0.0/skins/moono/
+third_party/static/ckeditor-bootstrapck-1.0.0/skins/ckbuilder.jar/
+third_party/static/ckeditor-bootstrapck-1.0.0/skins/bootstrapck/sample/
+third_party/static/ckeditor-bootstrapck-1.0.0/skins/bootstrapck/scss/
+third_party/static/fontawesome-free-5.9.0-web/
+third_party/static/guppy-7509f3/site/
+third_party/static/guppy-7509f3/test/
+third_party/static/MathJax-2.7.5/docs/
+third_party/static/MathJax-2.7.5/fonts/HTML-CSS/Gyre-Pagella/
+third_party/static/MathJax-2.7.5/fonts/HTML-CSS/Gyre-Termes/
+third_party/static/MathJax-2.7.5/fonts/HTML-CSS/Latin-Modern/
+third_party/static/MathJax-2.7.5/fonts/HTML-CSS/Neo-Euler/
+third_party/static/MathJax-2.7.5/fonts/HTML-CSS/TeX/png/
+third_party/static/MathJax-2.7.5/localization/ast/
+third_party/static/MathJax-2.7.5/localization/bcc/
+third_party/static/MathJax-2.7.5/localization/bg/
+third_party/static/MathJax-2.7.5/localization/br/
+third_party/static/MathJax-2.7.5/localization/ca/
+third_party/static/MathJax-2.7.5/localization/cdo/
+third_party/static/MathJax-2.7.5/localization/cs/
+third_party/static/MathJax-2.7.5/localization/cy/
+third_party/static/MathJax-2.7.5/localization/da/
+third_party/static/MathJax-2.7.5/localization/de/
+third_party/static/MathJax-2.7.5/localization/eo/
+third_party/static/MathJax-2.7.5/localization/es/
+third_party/static/MathJax-2.7.5/localization/fa/
+third_party/static/MathJax-2.7.5/localization/fi/
+third_party/static/MathJax-2.7.5/localization/fr/
+third_party/static/MathJax-2.7.5/localization/gl/
+third_party/static/MathJax-2.7.5/localization/he/
+third_party/static/MathJax-2.7.5/localization/ia/
+third_party/static/MathJax-2.7.5/localization/it/
+third_party/static/MathJax-2.7.5/localization/ja/
+third_party/static/MathJax-2.7.5/localization/kn/
+third_party/static/MathJax-2.7.5/localization/ko/
+third_party/static/MathJax-2.7.5/localization/lb/
+third_party/static/MathJax-2.7.5/localization/lki/
+third_party/static/MathJax-2.7.5/localization/lt/
+third_party/static/MathJax-2.7.5/localization/mk/
+third_party/static/MathJax-2.7.5/localization/nl/
+third_party/static/MathJax-2.7.5/localization/oc/
+third_party/static/MathJax-2.7.5/localization/pl/
+third_party/static/MathJax-2.7.5/localization/pt/
+third_party/static/MathJax-2.7.5/localization/pt-br/
+third_party/static/MathJax-2.7.5/localization/qqq/
+third_party/static/MathJax-2.7.5/localization/ru/
+third_party/static/MathJax-2.7.5/localization/scn/
+third_party/static/MathJax-2.7.5/localization/sco/
+third_party/static/MathJax-2.7.5/localization/sl/
+third_party/static/MathJax-2.7.5/localization/sv/
+third_party/static/MathJax-2.7.5/localization/tr/
+third_party/static/MathJax-2.7.5/localization/uk/
+third_party/static/MathJax-2.7.5/localization/vi/
+third_party/static/MathJax-2.7.5/localization/zh-hans/
+third_party/static/MathJax-2.7.5/test/
+third_party/static/MathJax-2.7.5/unpacked/
+third_party/static/messageformat-2.0.5/
+third_party/static/popperJs-1.15.0/

.github/CODEOWNERS

@@ -237,11 +237,9 @@
 
 
 # Google-app-engine configurations
-/appengine_config*.py @nithusha21
 /cron.yaml @nithusha21
 /index.yaml @vojtechjelinek
 /queue.yaml @nithusha21
-/main_taskqueue.py @nithusha21
 
 
 # Global components and filters.
@@ -460,16 +458,14 @@
 /core/templates/domain/jobs/ @brianrodri
 /core/controllers/cron*.py @nithusha21
 /core/domain/cron_services*.py @nithusha21
-/main_cron.py @nithusha21
-/main.py @DubeySandeep @nithusha21
+/main*.py @DubeySandeep @nithusha21
 /feconf.py @DubeySandeep @nithusha21 @seanlip
 /constants*.py @nithusha21
 /assets/constants.ts @nithusha21
 /core/controllers/tasks*.py @aks681
 /core/domain/beam_job*.py @brianrodri
 /core/domain/email*.py @aks681
 /core/domain/image_service*.py @DubeySandeep
-/core/jobs*.py @brianrodri
 /core/platform/ @brianrodri
 /core/templates/App*.ts @srijanreddy98
 /core/templates/app.constants.ts @srijanreddy98
@@ -526,6 +522,7 @@
 
 
 # Release team.
+/.gcloudignore @nithusha21
 /core/controllers/release_coordinator*.py @nithusha21 @vojtechjelinek
 /core/templates/pages/release-coordinator-page/ @nithusha21 @vojtechjelinek
 /scripts/release_scripts/ @nithusha21 @vojtechjelinek
@@ -602,8 +599,7 @@
 /core/templates/pages/header_js_libs.html @vojtechjelinek
 /core/templates/services/csrf-token.service*.ts @vojtechjelinek
 /core/templates/third-party-imports/ @nithusha21 @vojtechjelinek
-/.lighthouserc.js @jimbyo @vojtechjelinek
-/.lighthouserc-accessibility.js @jimbyo @vojtechjelinek
+/.lighthouserc*.js @jimbyo @vojtechjelinek
 /puppeteer-login-script.js @jimbyo @vojtechjelinek
 /scripts/run_lighthouse_tests.py @vojtechjelinek
 /webpack.*.ts @nithusha21 @vojtechjelinek

.github/workflows/backend_tests.yml

@@ -21,17 +21,11 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        if: startsWith(github.head_ref, 'update-changelog-for-release') == false
-        run: pip install wheel==0.35.0
       - name: Install third party
         if: startsWith(github.head_ref, 'update-changelog-for-release') == false
         run: python -m scripts.install_third_party_libs
-      - name: Install packages for optimized coverage
-        if: startsWith(github.head_ref, 'update-changelog-for-release') == false
-        run: sudo apt-get install -y python-dev gcc
       - name: Install coverage and configparser
         if: startsWith(github.head_ref, 'update-changelog-for-release') == false
         run: pip install coverage configparser
@@ -52,11 +46,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        if: startsWith(github.head_ref, 'update-changelog-for-release') == false
-        run: pip install wheel==0.35.0
       # We need this so that we can check coverage of autogenerated code
       - name: Install third party
         if: startsWith(github.head_ref, 'update-changelog-for-release') == false

.github/workflows/e2e_additional_editor_and_player.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs
@@ -40,6 +38,11 @@ jobs:
         run: xvfb-run -a --server-args="-screen 0, 1285x1000x24" python -m scripts.run_e2e_tests --suite="additionalEditorFeatures" --prod_env
         env: 
           VIDEO_RECORDING_IS_ENABLED: 0
+      - name: Run Additional Editor Modals E2E Test
+        if: ${{ steps.check-risk.outputs.is-low-risk != 0 }}
+        run: xvfb-run -a --server-args="-screen 0, 1285x1000x24" python -m scripts.run_e2e_tests --skip-build --skip-install --suite="additionalEditorFeaturesModals" --prod_env
+        env:
+          VIDEO_RECORDING_IS_ENABLED: 0
       - name: Run Additional Player E2E Test
         if: ${{ steps.check-risk.outputs.is-low-risk != 0 }}
         run: xvfb-run -a --server-args="-screen 0, 1285x1000x24" python -m scripts.run_e2e_tests --skip-build --skip-install --suite="additionalPlayerFeatures" --prod_env

.github/workflows/e2e_creator_learner_dashboard_and_editor_tabs.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs

.github/workflows/e2e_history_statistics_tabs_and_extensions.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs

.github/workflows/e2e_learner_flow_skill_editor_and_embedding.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs

.github/workflows/e2e_miscellaneous_tests.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs

.github/workflows/e2e_other_tests.yml

@@ -19,10 +19,8 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: '2.x'
+          python-version: '3.7'
           architecture: 'x64'
-      - name: Setup python by installing wheel
-        run: pip install wheel==0.35.0
       - uses: ./.github/actions/merge
       - name: Install third party
         run: python -m scripts.install_third_party_libs