Spackbot should notice failed style checks and suggest action

[scottwittenburg]

Spackbot registers for check_run events and when it receives such a notification corresponding to a failed style check, it is supposed to post a comment to the PR suggesting how to fix the problem. This is not happening.

The problem seems to be that many (or possibly all) check_run events contain an empty list for pull_requests, which is used here to identify the PR where the helpful comment should be posted.

In this community discussion comment, someone has reported that the pull_requests field is only non-empty on the very first check_run event triggered for a PR, but the docs for that event seem to state that the pull_requests field will always be empty for a pull request where the head is on a fork. Either way it doesn't seem a reliable way to know the PR with which the check is associated.

Below is an example payload I captured recently. I don't see any direct way to link the check_run event to a pull request based on the data in this payload. However it appears we might be able to make another request at this point to list the prs associated with a commit and find, among the returned list of pull requests, the one matching the head_sha in the check_run.

Example `check_run` payload

{
    'action': 'completed',
    'check_run': {
        'id': 7964523948,
        'name': 'style',
        'node_id': 'CR_kwDOFrPNBs8AAAAB2rj9rA',
        'head_sha': '42d8232f1b38584e2ac0df7d7027255486feae2a',
        'external_id': '3f85e25b-fb23-5eb7-8385-51966a33df73',
        'url': 'https://api.github.com/repos/spack-test/spack/check-runs/7964523948',
        'html_url': 'https://github.com/spack-test/spack/runs/7964523948?check_suite_focus=true',
        'details_url': 'https://github.com/spack-test/spack/runs/7964523948?check_suite_focus=true',
        'status': 'completed',
        'conclusion': 'failure',
        'started_at': '2022-08-23T01:52:11Z',
        'completed_at': '2022-08-23T01:54:16Z',
        'output': {
            'title': 'style',
            'summary': 'There are 1 failures, 0 warnings, and 0 notices.',
            'text': None,
            'annotations_count': 1,
            'annotations_url': 'https://api.github.com/repos/spack-test/spack/check-runs/7964523948/annotations'
        },
        'check_suite': {
            'id': 7932076708,
            'node_id': 'CS_kwDOFrPNBs8AAAAB2MnipA',
            'head_branch': 'wrong-order-imports',
            'head_sha': '42d8232f1b38584e2ac0df7d7027255486feae2a',
            'status': 'in_progress',
            'conclusion': None,
            'url': 'https://api.github.com/repos/spack-test/spack/check-suites/7932076708',
            'before': '68ff8c762f9cfa775fe45035cb905182dfcc77a5',
            'after': '42d8232f1b38584e2ac0df7d7027255486feae2a',
            'pull_requests': [],
            'app': {
                'id': 15368,
                'slug': 'github-actions',
                'node_id': 'MDM6QXBwMTUzNjg=',
                'owner': {
                    'login': 'github',
                    'id': 9919,
                    'node_id': 'MDEyOk9yZ2FuaXphdGlvbjk5MTk=',
                    'avatar_url': 'https://avatars.githubusercontent.com/u/9919?v=4',
                    'gravatar_id': '',
                    'url': 'https://api.github.com/users/github',
                    'html_url': 'https://github.com/github',
                    'followers_url': 'https://api.github.com/users/github/followers',
                    'following_url': 'https://api.github.com/users/github/following{/other_user}',
                    'gists_url': 'https://api.github.com/users/github/gists{/gist_id}',
                    'starred_url': 'https://api.github.com/users/github/starred{/owner}{/repo}',
                    'subscriptions_url': 'https://api.github.com/users/github/subscriptions',
                    'organizations_url': 'https://api.github.com/users/github/orgs',
                    'repos_url': 'https://api.github.com/users/github/repos',
                    'events_url': 'https://api.github.com/users/github/events{/privacy}',
                    'received_events_url': 'https://api.github.com/users/github/received_events',
                    'type': 'Organization',
                    'site_admin': False
                },
                'name': 'GitHub Actions',
                'description': 'Automate your workflow from idea to production',
                'external_url': 'https://help.github.com/en/actions',
                'html_url': 'https://github.com/apps/github-actions',
                'created_at': '2018-07-30T09:30:17Z',
                'updated_at': '2019-12-10T19:04:12Z',
                'permissions': {
                    'actions': 'write',
                    'administration': 'read',
                    'checks': 'write',
                    'contents': 'write',
                    'deployments': 'write',
                    'discussions': 'write',
                    'issues': 'write',
                    'merge_queues': 'write',
                    'metadata': 'read',
                    'packages': 'write',
                    'pages': 'write',
                    'pull_requests': 'write',
                    'repository_hooks': 'write',
                    'repository_projects': 'write',
                    'security_events': 'write',
                    'statuses': 'write',
                    'vulnerability_alerts': 'read'
                },
                'events': ['branch_protection_rule', 'check_run', 'check_suite', 'create', 'delete', 'deployment', 'deployment_status', 'discussion', 'discussion_comment', 'fork', 'gollum', 'issues', 'issue_comment', 'label', 'merge_group', 'milestone', 'page_build', 'project', 'project_card', 'project_column', 'public', 'pull_request', 'pull_request_review', 'pull_request_review_comment', 'push', 'registry_package', 'release', 'repository', 'repository_dispatch', 'status', 'watch', 'workflow_dispatch', 'workflow_run']
            },
            'created_at': '2022-08-23T01:51:38Z',
            'updated_at': '2022-08-23T01:52:11Z'
        },
        'app': {
            'id': 15368,
            'slug': 'github-actions',
            'node_id': 'MDM6QXBwMTUzNjg=',
            'owner': {
                'login': 'github',
                'id': 9919,
                'node_id': 'MDEyOk9yZ2FuaXphdGlvbjk5MTk=',
                'avatar_url': 'https://avatars.githubusercontent.com/u/9919?v=4',
                'gravatar_id': '',
                'url': 'https://api.github.com/users/github',
                'html_url': 'https://github.com/github',
                'followers_url': 'https://api.github.com/users/github/followers',
                'following_url': 'https://api.github.com/users/github/following{/other_user}',
                'gists_url': 'https://api.github.com/users/github/gists{/gist_id}',
                'starred_url': 'https://api.github.com/users/github/starred{/owner}{/repo}',
                'subscriptions_url': 'https://api.github.com/users/github/subscriptions',
                'organizations_url': 'https://api.github.com/users/github/orgs',
                'repos_url': 'https://api.github.com/users/github/repos',
                'events_url': 'https://api.github.com/users/github/events{/privacy}',
                'received_events_url': 'https://api.github.com/users/github/received_events',
                'type': 'Organization',
                'site_admin': False
            },
            'name': 'GitHub Actions',
            'description': 'Automate your workflow from idea to production',
            'external_url': 'https://help.github.com/en/actions',
            'html_url': 'https://github.com/apps/github-actions',
            'created_at': '2018-07-30T09:30:17Z',
            'updated_at': '2019-12-10T19:04:12Z',
            'permissions': {
                'actions': 'write',
                'administration': 'read',
                'checks': 'write',
                'contents': 'write',
                'deployments': 'write',
                'discussions': 'write',
                'issues': 'write',
                'merge_queues': 'write',
                'metadata': 'read',
                'packages': 'write',
                'pages': 'write',
                'pull_requests': 'write',
                'repository_hooks': 'write',
                'repository_projects': 'write',
                'security_events': 'write',
                'statuses': 'write',
                'vulnerability_alerts': 'read'
            },
            'events': ['branch_protection_rule', 'check_run', 'check_suite', 'create', 'delete', 'deployment', 'deployment_status', 'discussion', 'discussion_comment', 'fork', 'gollum', 'issues', 'issue_comment', 'label', 'merge_group', 'milestone', 'page_build', 'project', 'project_card', 'project_column', 'public', 'pull_request', 'pull_request_review', 'pull_request_review_comment', 'push', 'registry_package', 'release', 'repository', 'repository_dispatch', 'status', 'watch', 'workflow_dispatch', 'workflow_run']
        },
        'pull_requests': []
    },
    'repository': {
        'id': 380882182,
        'node_id': 'MDEwOlJlcG9zaXRvcnkzODA4ODIxODI=',
        'name': 'spack',
        'full_name': 'spack-test/spack',
        'private': False,
        'owner': {
            'login': 'spack-test',
            'id': 86581670,
            'node_id': 'MDEyOk9yZ2FuaXphdGlvbjg2NTgxNjcw',
            'avatar_url': 'https://avatars.githubusercontent.com/u/86581670?v=4',
            'gravatar_id': '',
            'url': 'https://api.github.com/users/spack-test',
            'html_url': 'https://github.com/spack-test',
            'followers_url': 'https://api.github.com/users/spack-test/followers',
            'following_url': 'https://api.github.com/users/spack-test/following{/other_user}',
            'gists_url': 'https://api.github.com/users/spack-test/gists{/gist_id}',
            'starred_url': 'https://api.github.com/users/spack-test/starred{/owner}{/repo}',
            'subscriptions_url': 'https://api.github.com/users/spack-test/subscriptions',
            'organizations_url': 'https://api.github.com/users/spack-test/orgs',
            'repos_url': 'https://api.github.com/users/spack-test/repos',
            'events_url': 'https://api.github.com/users/spack-test/events{/privacy}',
            'received_events_url': 'https://api.github.com/users/spack-test/received_events',
            'type': 'Organization',
            'site_admin': False
        },
        'html_url': 'https://github.com/spack-test/spack',
        'description': 'A flexible package manager that supports multiple versions, configurations, platforms, and compilers.',
        'fork': True,
        'url': 'https://api.github.com/repos/spack-test/spack',
        'forks_url': 'https://api.github.com/repos/spack-test/spack/forks',
        'keys_url': 'https://api.github.com/repos/spack-test/spack/keys{/key_id}',
        'collaborators_url': 'https://api.github.com/repos/spack-test/spack/collaborators{/collaborator}',
        'teams_url': 'https://api.github.com/repos/spack-test/spack/teams',
        'hooks_url': 'https://api.github.com/repos/spack-test/spack/hooks',
        'issue_events_url': 'https://api.github.com/repos/spack-test/spack/issues/events{/number}',
        'events_url': 'https://api.github.com/repos/spack-test/spack/events',
        'assignees_url': 'https://api.github.com/repos/spack-test/spack/assignees{/user}',
        'branches_url': 'https://api.github.com/repos/spack-test/spack/branches{/branch}',
        'tags_url': 'https://api.github.com/repos/spack-test/spack/tags',
        'blobs_url': 'https://api.github.com/repos/spack-test/spack/git/blobs{/sha}',
        'git_tags_url': 'https://api.github.com/repos/spack-test/spack/git/tags{/sha}',
        'git_refs_url': 'https://api.github.com/repos/spack-test/spack/git/refs{/sha}',
        'trees_url': 'https://api.github.com/repos/spack-test/spack/git/trees{/sha}',
        'statuses_url': 'https://api.github.com/repos/spack-test/spack/statuses/{sha}',
        'languages_url': 'https://api.github.com/repos/spack-test/spack/languages',
        'stargazers_url': 'https://api.github.com/repos/spack-test/spack/stargazers',
        'contributors_url': 'https://api.github.com/repos/spack-test/spack/contributors',
        'subscribers_url': 'https://api.github.com/repos/spack-test/spack/subscribers',
        'subscription_url': 'https://api.github.com/repos/spack-test/spack/subscription',
        'commits_url': 'https://api.github.com/repos/spack-test/spack/commits{/sha}',
        'git_commits_url': 'https://api.github.com/repos/spack-test/spack/git/commits{/sha}',
        'comments_url': 'https://api.github.com/repos/spack-test/spack/comments{/number}',
        'issue_comment_url': 'https://api.github.com/repos/spack-test/spack/issues/comments{/number}',
        'contents_url': 'https://api.github.com/repos/spack-test/spack/contents/{+path}',
        'compare_url': 'https://api.github.com/repos/spack-test/spack/compare/{base}...{head}',
        'merges_url': 'https://api.github.com/repos/spack-test/spack/merges',
        'archive_url': 'https://api.github.com/repos/spack-test/spack/{archive_format}{/ref}',
        'downloads_url': 'https://api.github.com/repos/spack-test/spack/downloads',
        'issues_url': 'https://api.github.com/repos/spack-test/spack/issues{/number}',
        'pulls_url': 'https://api.github.com/repos/spack-test/spack/pulls{/number}',
        'milestones_url': 'https://api.github.com/repos/spack-test/spack/milestones{/number}',
        'notifications_url': 'https://api.github.com/repos/spack-test/spack/notifications{?since,all,participating}',
        'labels_url': 'https://api.github.com/repos/spack-test/spack/labels{/name}',
        'releases_url': 'https://api.github.com/repos/spack-test/spack/releases{/id}',
        'deployments_url': 'https://api.github.com/repos/spack-test/spack/deployments',
        'created_at': '2021-06-28T02:20:07Z',
        'updated_at': '2021-10-02T03:47:08Z',
        'pushed_at': '2022-08-23T01:51:35Z',
        'git_url': 'git://github.com/spack-test/spack.git',
        'ssh_url': '[email protected]:spack-test/spack.git',
        'clone_url': 'https://github.com/spack-test/spack.git',
        'svn_url': 'https://github.com/spack-test/spack',
        'homepage': 'https://spack.io',
        'size': 188002,
        'stargazers_count': 1,
        'watchers_count': 1,
        'language': 'Python',
        'has_issues': False,
        'has_projects': True,
        'has_downloads': True,
        'has_wiki': True,
        'has_pages': False,
        'forks_count': 4,
        'mirror_url': None,
        'archived': False,
        'disabled': False,
        'open_issues_count': 6,
        'license': {
            'key': 'other',
            'name': 'Other',
            'spdx_id': 'NOASSERTION',
            'url': None,
            'node_id': 'MDc6TGljZW5zZTA='
        },
        'allow_forking': True,
        'is_template': False,
        'web_commit_signoff_required': False,
        'topics': [],
        'visibility': 'public',
        'forks': 4,
        'open_issues': 6,
        'watchers': 1,
        'default_branch': 'develop'
    },
    'organization': {
        'login': 'spack-test',
        'id': 86581670,
        'node_id': 'MDEyOk9yZ2FuaXphdGlvbjg2NTgxNjcw',
        'url': 'https://api.github.com/orgs/spack-test',
        'repos_url': 'https://api.github.com/orgs/spack-test/repos',
        'events_url': 'https://api.github.com/orgs/spack-test/events',
        'hooks_url': 'https://api.github.com/orgs/spack-test/hooks',
        'issues_url': 'https://api.github.com/orgs/spack-test/issues',
        'members_url': 'https://api.github.com/orgs/spack-test/members{/member}',
        'public_members_url': 'https://api.github.com/orgs/spack-test/public_members{/member}',
        'avatar_url': 'https://avatars.githubusercontent.com/u/86581670?v=4',
        'description': None
    },
    'sender': {
        'login': 'prmsgbot',
        'id': 31325305,
        'node_id': 'MDQ6VXNlcjMxMzI1MzA1',
        'avatar_url': 'https://avatars.githubusercontent.com/u/31325305?v=4',
        'gravatar_id': '',
        'url': 'https://api.github.com/users/prmsgbot',
        'html_url': 'https://github.com/prmsgbot',
        'followers_url': 'https://api.github.com/users/prmsgbot/followers',
        'following_url': 'https://api.github.com/users/prmsgbot/following{/other_user}',
        'gists_url': 'https://api.github.com/users/prmsgbot/gists{/gist_id}',
        'starred_url': 'https://api.github.com/users/prmsgbot/starred{/owner}{/repo}',
        'subscriptions_url': 'https://api.github.com/users/prmsgbot/subscriptions',
        'organizations_url': 'https://api.github.com/users/prmsgbot/orgs',
        'repos_url': 'https://api.github.com/users/prmsgbot/repos',
        'events_url': 'https://api.github.com/users/prmsgbot/events{/privacy}',
        'received_events_url': 'https://api.github.com/users/prmsgbot/received_events',
        'type': 'User',
        'site_admin': False
    },
    'installation': {
        'id': 27201021,
        'node_id': 'MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMjcyMDEwMjE='
    }
}