What would improve your CI experience?

[AshCripps]

refs: nodejs/build#2550

We at @nodejs/build exploring whether there would be any interest in establishing a "Build Helper" role. The overall gist of this role would be to open up certain parts of build to collaborators to enable them to solve some minor CI issues they face which may be blocking PRs without waiting for a member of the build team to respond.

Examples come to mind are - Being able to clear the workspace on a machine that has run out of disk space and is preventing new jobs from being spun up (A common fault on the nearform- macs) or being able to make certain nodes as flaky/offline and leaving for a build member to inspect and resolve later.

So Im opening this discussion for people to give their opinion on what common issues they face with CI, what issues they would be comfortable solving themselves and whether they have ideas on what we at build can provide to make your lives easier.

[MylesBorins]

@AshCripps are you imagining that folks would have some scripts or dashboard they could use to do this maintenance or would we be giving people full access to the test machines?

[AshCripps]

Im toying with idea of either - I like the idea of a dashboard or something where people can authenticate to and run the scripts without having to know what each command does etc. It would definitely be more work upfront but would have a better payoff imo. With the dashboard approach we can really lock down what can be done so that alleviates some of the trust pressure we put on build resources.

[MylesBorins]

If we can make a good dashboard we could arguably make it available to anyone with a commit bit

[mhdawson]

We are already relatively liberal with access to the test machines, so I think if necessary giving full access should be considered as well.

[AshCripps]

Revisting this.

Seems to be the best two options are:

1: build a "system" of sorts that allows a user to authenticate (via github) into a dashboard from which they can run a series of diagnostic/system functions. e.g. restart jenkins agent, clear workspace etc.

2: Create a clone of @nodejs/build-test the lowest tier of build membership which people can be added too - this means they can log into all the test machines and fix issues that way without being hit the barrage of mentions @nodejs/build gets. For this to work it would require some robust documentation drawn up to ensure that even the least confident person should be able to ran basic tasks without fear of breaking something.

Personally I like the idea of 1 but am fully aware its a lot of work upfront and would need to time to just spec out but the end result would be much cleaner than 2 which would require people to use the secrets system which can be pretty confusing imo.

Thoughts?

[BethGriggs]

For 1, I'm curious if it'd be possible to create a 'CI Helper' or 'CI Maintenance' tab (or whatever name fits) in Jenkins with parameterised jobs that solve the most common issues such as force wiping workspaces or killing processes on a specified machine, purging CDN, etc. (As a releaser, it'd be great to just be able to run a job that purges the CDN without needing direct access to the machine.)

Appreciate that those jobs would need elevated access to other machines by some mechanism(?) - so should those jobs should still be restricted, but maybe could be available to a wider subset than @nodejs/build.

[mhdawson]

Exploring ideas for how we give more access is good.

I'm wondering if we should pick 1 (or a short list) of things we would like build helpers to be able to do and then discuss how we let them do that in the context of specific tasks.

[choilmto]

I'm interested in the being build helper, and I'd like to put my name down for the next opportunity for the role.