'Run internal testing via comment triggers' - rationale and workflow:

As a first step, an 'Org contact user' compiles a list of Org-approved user names and a unique Org comment 'trigger string'. The trigger string should conform to user name format and length: alphanumeric characters & dashes, 39 chars.

• thereafter usage for all Org-approved-user comment triggers is: /trigger-string

User and trigger information is then added to a repository registry (currently repo file: .github/workflows/internal_test_list) where users and comment triggers are stored as follows:

/user1//trigger1 /user2//trigger1
/user3//trigger2 /user4//trigger2
etc

• Each entry is space(s) or newline(s) separated.
• The list should be periodically audited to confirm that contents are still required.

On merge of the updated list to the mainline branch, each listed user is registered as an 'internal-test requester' and is bound to a specific Org comment trigger e.g. '/trigger2' above. Via this mechanism, External-side users can request Internal-side test runs.

Org contact user should then define a repo access token allowing:

1. scanning of PR comments,
2. PR comment-body reads and
3. PR comment writes.

Note that 'fine-grain' user-PATs allow the setting of various individual 'repository permissions' including read/write "pull request" perms (pull requests and related comments, assignees, labels, milestones, and merges). These offer the Internal-side granularity and permissons required (and are used in our own Internal-side implementation).

When a registered Org user makes a request to run an internal-test by entering the comment trigger ("/trigger-string") as a PR comment body our 'internal-test-request comment' trigger fires. It checks the following:

• its a new comment (no edits or deletes).
• its a PR (and not an Issue - Github internal quirk).
• the comment has the correct '/trigger_string' format.
• the Org user exists in the internal_test_list registry and the comment trigger used is assigned to that user.
• the PR is open.

If any check fails the trigger stops and any remaining trigger action steps and jobs will be silently skipped. The action itself will never fail.

If all checks pass the External-side 'github-actions' bot then issues a validated 'internal-test request' as a comment which includes a request-data body in the following format:

/trigger-string
<PR reference> (commit SHA - which, again by Github quirk, is not simple to find.)
<validated requester @user_name> (e.g. @user1)
Codeplay Internal Testing: Request Validated

e.g. of External-side "validation" comment:

/verify 584e585 @alan-forbes-cp Codeplay Internal Testing: Request Validated

Using the generated access token, the Internal-side bot then scans for such comments, runs internal tests and reports final pass/fail to the External-side PR as a comment.

• Suggested 'best practice' for the Internal-side is to respond both with a "Started" comment and with a final "Completed: PASS|FAIL|DID NOT RUN|TIMED OUT|etc." comment, adopting the format used by the External-side. As a bare minimum, Internal-side should indicate test completion as a handshake between the sides.
• Note that Internal-side comments are not validated or policed by the External-side.

e.g. of Internal-side comments:

/verify 584e585 @alan-forbes-cp Codeplay Internal Testing: Started
/verify 584e585 @alan-forbes-cp Codeplay Internal Testing: Completed: PASS

Notes:

• Comments in PRs and Issues are "the same but different" so its necessary to distinguish between the two in the action code.
• Only trigger actions contained in the mainline workflow files are used (they run "from the mainline" which partly explains why it can be tricky to get PR branch information e.g. PR branch name and commit SHA). Similarly, only the mainline internal_test_list registry is referenced by the trigger action.
• Currently our action requires that any runner handling our trigger has the Github "gh" cli tool installed - however all (non-self-hosted) runners will have this.
• Formal internal-test request data (i.e. External-side bot comment text) can be tailored for Internal-side handshaking reqs as required.
• Security-related aspects for review include:
  • use of vanilla 'github.token'
  • use of the extra "pull_requests:write" permission in action
  • use of env vars in action's bash scripting
  • use of an explicit commit hash with the 'checkout' action
  • auth perms for Internal-side bots?
  • anything else?