Framework: Automating Issue / Pull Request Maintenance and Cleanup

[william76]

Description

Trilinos currently has over 1100 open issues and over 40 open pull requests. We have some open issues that date back to 2015 with no activity since 2017, and PR's that have also had no activity in over a year. This becomes difficult to manage from the framework side because it's hard to tell what issues have been abandoned and which ones should be kept around for some reason such as planning, documentation, etc.

This issue is to facilitate some discussion on my proposal to set up a GitHub Action which will automatically mark issues stale and close them out after long term inactivity.

Proposal

I'd like to set up a GitHub Action to flag issues and pull requests as 'stale' and then close them based on the following rules:

1. Issues or Pull Requests with no activity for 12 months will get a new flag: FLAGGED FOR AUTO CLOSE
2. If the Issue or PR has had no new activity after being flagged in (1) after another month, then the issue or PR is closed by the bot.
3. When being flagged or closed the bot can add a useful message to the comments explaining why it's being autoclosed.
4. Issues or PR's that we wish to be exempt from the bot can be given a label that would exempt it from this check.
   1. NO AUTOCLOSE - This could be a generic flag to exempt items from the bot.
   2. DOCUMENTATION - This could be something to track Issues that are used for tracking documentation.
   3. PROJECT PLANNING - This could be added for issues used for long term project planning type things.
   4. CLOSED DUE TO INACTIVITY - A label to apply to issues and PR's that were autoclosed by the bot (makes them more searchable).

The flag names here are mostly just a quick stab at what we might be able to do. The main page for the particular runner we're looking at is here. It has quite a few control options. We can set a custom message that it will add to the comments for Issues or PR's when it's flagging them and when its closing them. We can also add in a flag that would be applied to issues that are closed via the bot which might be useful if we need to dig through what was autoclosed.

I think we can iterate on what Labels we might want to use to exempt things from the bot and how long we might want to give before marking stuff for removal. I had thought 1 year to flagging and then an additional month before closing since we do have things that stick around long term in Trilinos so enforcing a policy that people should touch issues and PR's at least once a year if they wish to keep them around isn't a huge ask. Adding in the ability to explicitly exempt items from the bot also benefits us because those issues would become more easily searched.

I have a test repository to play around with the tool that marks things stale after 1 day and closes them after another day if they remain inactive. If you're interested, the repo is here with this example issue that shows some of the messages and flags, etc.

Let us know what you guys think. :D

Interested People

@mentioning folks from today's planning meeting that seemed engaged in this issue. Feel free to tag others you think who might be interested.
@ccober6
@srajama1
@maherou
@csiefer2
@kddevin
@bartlettroscoe
@jwillenbring
@prwolfe
@trilinos/framework

[csiefer2]

Before we start debating software solutions, the fundamental question we need to ask ourselves is: What are issues for?

Some answers may include:

1. Tracking actual user bugs / feature requests.
2. Answering user questions about using Trilinos.
3. Tracking work by a single developer on a project.
4. Coordinating work across multiple developers on a task.
5. Discussing design.
6. Recording bugs, gotchas and other "we should really fix this" information for the future.

And there are probably more! These different types of information should likely have different expectations for how long issues should last. Maybe they should all be on github either.

[kddevin]

Github issues with @csiefer2's ideas 1-6 above provide helpful artifacts for software quality assessments of Trilinos packages. I am glad to have these things archived in github.
Do merged PRs automatically close issues if they correctly reference the issue? If not, that would be a nice feature.

[bartlettroscoe]

If not, that would be a nice feature.

@kddevin, I think it will if you put Fixes #<issue-id> in a comment in the PR. See here.

The bigger problem that there are a lot of PRs that don't even mention the GitHub Issue they are supposed to address.

[william76]

This is the list of Labels that we can add to Trilinos that are useful to the auto closer tool.

Label	Issue?	PR?	Description
MARKED_FOR_CLOSURE	X	X	Issue or PR is marked for auto-closure due to inactivity.
CLOSED_DUE_TO_INACTIVITY	X	X	Issue or PR was automatically closed due to inactivity.
DO_NOT_AUTOCLOSE	X		This issue or PR should not be automatically closed due to inactivity.

Some issues are used by Trilinos developers for long-term knowledge capture and archival, but we don't necessarily need to keep these issues open. As a reminder, GitHub does not delete information and closing an issue does not delete it. One can remove the is:open filter to remove the restriction to only open issues.

In my draft rules, I believe we should not allow exemption of Pull Requests from auto-closure. Given how the Trilinos code changes over time due to development code changes in a PR can become outdated fairly quickly. I think it's reasonable that any open Pull Request should require some intervention by the interested parties at least once every 395 days to keep it open. This only requires some activity on the PR to occur -- adding a comment, setting/removing a label or additional pushes on the source branch/fork (I think). If a PR is part of long term work and won't be worked on for a while it is better to track the activity in an issue and link to the PR, which will keep any relevant PR's associated with the issue regardless of the open/closed state of the PR.

For deeper organization, we can add new Labels to Trilinos for developers to tag issues to categorize them. For example:

Label	Description
DOCUMENTATION	Issue is used as documentation
PLANNING	Issue captures planning information

Adding these could be used to further filter down the issues using advanced filters. For example, to search for open issues with labels for both Tpetra and KokkosKernels you could use this filter: is:issue is:open label:"pkg: Tpetra" label:"pkg: KokkosKernels".

I do not want to complicate the runner tool with a large list of flags that would 'exempt' issues and PR's, keeping us at 1 flag keeps things more straightforward. We can add additional labels to Trilinos for categorization purposes as needed.

Comment Messages for the Github Action

The following message snippets can be used with the auto-closer:

Inactive Issue Message

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.

If you would like to keep this issue open please add a comment and remove the MARKED_FOR_CLOSURE label.

If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.

Closed Issue Message

This issue was closed due to inactivity for 395 days.

Inactive PR Message

This Pull Request has been marked for closure due to inactivity.

Because of the changing nature of the Trilinos source due to active development, a pull request with no activity for 365* days is considered to be abandoned and will be automatically closed after 395 additional days of inactivity from when it was marked inactive.

If this should be kept open, please post a comment and remove the label MARKED_FOR_CLOSURE to reset the inactivity timer.

Closed PR Message

This Pull Request has been automatically closed due to 395 days of inactivity.

Action Items

• Generate a new Wiki page under Managing Issues on the Trilinos Wiki, "Handling Inactive Issues and Pull Requests" (draft is here: https://github.com/trilinos/Trilinos/wiki/Handling-Stale-Issues-and-Pull-Requests)
• Test out the auto-closer on a test repository
• Implement runner in dry-run mode to Trilinos repository to test some of the features and see what it will do.
• Seek to enable the action on December 10th. This would result in the first issues and PR's auto-closure date hit around January 10th, once the winter shutdown has completed.

FYI @jwillenbring

[jwillenbring]

I think we can close this. @william76 if so, please do so. Thanks.

[bartlettroscoe]

@william76, it was suggested that it might be good to mark issues that are auto-closed with the label "wortfix" to make it clear that that is what this means.