[GH Request] Requirements bot unable to create PRs in course-discovery

[DawoudSheraz]

Firm Name

2U

Urgency

Medium (< 2 weeks)

Requested Change

The upgrade requirements job in course-discovery has been failing on "Creating Pull Request" step with the following error:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/runner/work/course-discovery/course-discovery/testeng-ci/jenkins/pull_request_creator.py", line 240, in <module>
    main(auto_envvar_prefix="PR_CREATOR")  # pylint: disable=no-value-for-parameter
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/runner/work/course-discovery/course-discovery/testeng-ci/jenkins/pull_request_creator.py", line [23](https://github.com/openedx/course-discovery/actions/runs/6216696757/job/16870855327#step:8:24)6, in main
    creator.create(delete_old_pull_requests, untracked_files_required)
  File "/home/runner/work/course-discovery/course-discovery/testeng-ci/jenkins/pull_request_creator.py", line 148, in create
    self._create_new_branch()
  File "/home/runner/work/course-discovery/course-discovery/testeng-ci/jenkins/pull_request_creator.py", line 69, in _create_new_branch
    commit_sha = self.github_helper.update_list_of_files(
  File "/home/runner/work/course-discovery/course-discovery/testeng-ci/jenkins/github_helpers.py", line 4[34](https://github.com/openedx/course-discovery/actions/runs/6216696757/job/16870855327#step:8:35), in update_list_of_files
    new_git_tree = repository.create_git_tree(input_trees_list, base_tree=base_git_tree)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Repository.py", line 12[38](https://github.com/openedx/course-discovery/actions/runs/6216696757/job/16870855327#step:8:39), in create_git_tree
    headers, data = self._requester.requestJsonAndCheck(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 442, in requestJsonAndCheck
    return self.__check(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/github/Requester.py", line 487, in __check
    raise self.__createException(status, responseHeaders, data)
github.GithubException.UnknownObjectException: [40](https://github.com/openedx/course-discovery/actions/runs/6216696757/job/16870855327#step:8:41)4 {"message": "Not Found", "documentation_url": "https://docs.github.com/rest/git/trees#create-a-tree"}

This started happening about a month ago. At first, it was thought that it was a momentary issue but it has persisted. Our team suspects that this has something to do with the permissions of requirements-bot's access token for course-discovery.

• https://github.com/openedx/course-discovery/actions/runs/6216696757/job/16870855327
• https://github.com/openedx/course-discovery/actions/workflows/requirements-upgrade.yml

Reasoning

Upon discussing this issue with @UsamaSadiq from Arbi-bom, he mentioned that token updates could have caused this issue. Nothing is wrong with the actual make upgrade job and it is more of a permission issue. I am unclear what permission changes were made by Axim, if any. I am filing this ticket to understand if it is indeed the token's permission issue. Thank you

[openedx-workflow-automation]

Thank you for your report! @openedx/axim-oncall will triage within a business day. Simple requests usually take 2-3 business days to resolve; more complex requests could take longer.

[brian-smith-tcril]

Thanks for submitting this request! I'll look into this today and report back with my findings.

[brian-smith-tcril]

Considering the error is a 404, I'm not sure if this being a permissions issue makes sense. Based on the docs from the error I'd think something is causing the request to be sent to the wrong endpoint URL.

Can you verify the URL being used for the POST is correct?

[DawoudSheraz]

Sure @brian-smith-tcril I will double check this.

[brian-smith-tcril]

@DawoudSheraz I looked into the instructions here and found the edx-requirements-bot team did not have write access on the repo. I have added that. If you can test and see if that fixed the issue that'd be perfect!

[DawoudSheraz]

Hello @brian-smith-tcril , the latest run is green https://github.com/openedx/course-discovery/actions/runs/6303040340/job/17111476617. Adding the permission has fixed the issue. Thank you.