QA Team #45
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| - Title: qa-team | ||
| - Authors: [joltz](mailto:[email protected]) | ||
| - Start date: Feb 12 2020 | ||
| - RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) | ||
| - Tracking issue: [Edit if merged with link to tracking github issue] | ||
|
|
||
| --- | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| * The QA team for Grin: | ||
| * works to ensure that a high standard of quality for contributions is upheld across the project | ||
| * develops tools and processes to help monitor and enforce the quality of work | ||
| * exists to make sure that changes to the project are quality, not to produce the quality changes themselves | ||
| * ensures that contributors are encouraged, supported and empowered to produce the best quality work to be integrated into Grin | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| Over time, with many people contributing to many different areas of Grin, some things can get lost in the mix including quality. It could be beneficial to have a team supported by members, processes and tools to help provide a consistent level of quality contributions. | ||
|
|
||
| * Ensure that contributions to Grin are high quality | ||
| * Empower contributors to make high quality contributions to Grin | ||
| * Provide feedback from a QA perspective to micro and macro changes in Grin | ||
|
|
||
| # Community-level explanation | ||
| [community-level-explanation]: #community-level-explanation | ||
|
|
||
| The QA team observes micro and macro changes occurring over time with various contributions to Grin. They help review PRs, have opinions on both development and governance decisions and provide tooling frameworks for contributors to use and maintain to ensure a consistent high level of quality of work across the Grin project. The QA team provides support for contributors to encourage them to continue to submit quality work. | ||
|
|
||
| The day to day impact is more resources to empower contributors to make successful contributions to Grin by performing quality assurance testing and review on their own work. The long term impact is more consistency in the level of quality work that makes it into the Grin project over time. | ||
|
|
||
| ## Example | ||
|
|
||
| In this example a large PR is submitted that makes a nontrivial change to the Grin node. | ||
| * Before submitting the PR, the contributor: | ||
| * reads and integrates the well-formed contribution documentation to assist and support their efforts | ||
| * uses available testing frameworks to test their work | ||
| * Reviewers of the PR may use the documentation and tooling support provided by the QA team to perform another round of quality assurance testing and review on the submission | ||
| * Any necessary feedback is shared with the contributor to empower them to continue to make high quality contributions | ||
|
|
||
| # Reference-level explanation | ||
| [reference-level-explanation]: #reference-level-explanation | ||
|
|
||
| ## Membership | ||
|
|
||
| * Membership is bootstrapped by @lehnberg and @j01tz | ||
| * New membership approved with consensus of existing team members | ||
| * Members may be removed by self-resignation, consensus decision by QA team members or by core team | ||
|
|
||
| ## Decision Making | ||
|
|
||
| * A consensus-seeking approach is used for decision making through dialogue and discussion, modeled after the general principle in the Grin governance RFC[1]. | ||
| * In the event the team cannot reach consensus on a required action, the core team will determine the outcome | ||
|
|
||
| ## Responsibilities | ||
lehnberg marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| The responsibilities of the QA team _do_ include: | ||
| * Establishing and providing the necessary support for contributors to have a clear path to success by empowering them to perform quality assurance testing and review on their own work | ||
| * Clarifying, monitoring and enforcing the following of best QA practices by contributors | ||
|
|
||
| The responsibilities of the QA team _do not_ include: | ||
| * Doing the quality assurance testing and review work themselves | ||
| * Writing tests for code submissions and changes | ||
| * Keeping testing frameworks updated based on codebase changes | ||
| * Manually reviewing each submission for quality assurance | ||
| * _Was the codebase changed? Then the tests must also be changed or extended for full coverage- this is not the responsibility of the QA team!_ | ||
|
|
||
| ## Community Contributions | ||
|
|
||
| The QA team is ultimately responsible for creating an environment that encourages and empowers contributors to make quality contributions to Grin. To this end, the QA team will strive to provide the resources necessary for contributors to make consistent, high quality contributions to Grin, including but not limited to: documentation, constructive feedback, tooling and clarity of expectations of quality. | ||
|
There was a problem hiding this comment. This seems a little bit at odds with what's stated above about what the responsibilities do not include... for instance, surely the QA team will want to ensure their tooling and testing frameworks are up to date? Perhaps they won't be expected to maintain them 100% themselves, but the team would be the one striving to ensure they're in place, up-to-date and effective There was a problem hiding this comment. As it is currently written, my understanding is that the QA team is responsible for ensuring the tooling and frameworks are working before a contributor starts (this should be a function of the initial setup and ensuring that contributors don't make changes without adding working test coverage). The distinction is that the QA team is not responsible for updating the test framework to cover a change made by a contributor- the QA team is responsible for making sure that the contributor updates the test framework to cover their own change. This is supported by the QA team ensuring that expectations are documented, that the initial framework was set up properly and that changes from there were covered by their contributors. |
||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| * Resources are already spread thin, don't want to assign responsibilities if people won't have time | ||
|
|
||
| # Rationale and alternatives | ||
| [rationale-and-alternatives]: #rationale-and-alternatives | ||
|
|
||
| The alternative is a volunteer based approach to QA. This often leaves areas not properly assessed for quality that volunteers do not find interesting are already too busy making or reviewing other contributions. A volunteer based QA approach does not establish any accountability or oversight to changes in quality over time, nor does it provide active efforts to support new contributors. | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| The QA team RFC is following the guidance of the establishment of teams in the governance RFC[0]. | ||
|
|
||
| # Unresolved questions | ||
| [unresolved-questions]: #unresolved-questions | ||
|
|
||
| * To what extent does the QA team develop tools? | ||
|
There was a problem hiding this comment. This part of it is key to me. In my mind an effective QA takes a very active role in the creation of tools, frameworks and environments that continually deploy and test changes. Of course this can be supported by developers, but the QA team very much needs to drive the overall testing vision and take the leadership to ensure the correct tools are in place. There was a problem hiding this comment. I think it would be reasonable to expect the QA team to develop and make available the tools for sufficient testing. This looks like a big initial effort to get a test harness to support the node, wallet and miner. Additionally if new aspects emerge that require integration, an initial effort may be required to glue the necessary pieces so that contributors can begin adding and testing those as well. Beyond those initial efforts to add support to unsupported areas, as currently written, any changes should really be the responsibility of contributors. However if the QA team doesn't give contributors everything possible to succeed (good documentation, sane frameworks, stationary goalposts etc.) then the responsibility comes back to the QA team in my opinion. There's probably room for a lot of discussion and clarification here. For an ideal QA team, technical vision and leadership is essential. Whether this is necessary for a minimally viable QA team seems open (as is the question of whether a minimally viable QA team is worth it). |
||
| * What is the priority of quality in a contentious change with competing factors like core opinion, community opinion and security? | ||
| * How can we best maintain a balance of strictness of quality and openness to all levels of contributions and contributors? | ||
|
|
||
| # Future possibilities | ||
| [future-possibilities]: #future-possibilities | ||
|
|
||
| * Full, robust testing support for all areas of Grin | ||
| * Improved processes to support contributors | ||
|
|
||
| # References | ||
| [references]: #references | ||
|
|
||
| [0] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0002-grin-governance.md#teams | ||
| [1] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0002-grin-governance.md#general-principles | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering to what extent non-developer members of the QA team will be expected to fully understand the implications of major changes, particularly to the point where they're expected to help review PRs. Is it more a case of the QA team ensuring that tests exist or at least a testing strategy exists for verifying changes in PRs rather than the entire substance of the change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this point is important to discuss as test coverage can be a bit subjective and the inability to fully understand the changes could result in QA approving coverage of a feature that isn't properly tested. My initial reaction is that the QA team is there to stand up processes, protocols and tools and ensure that they are properly used by contributors. As indicated, development knowledge is required to assess whether the test coverage is accurate (not to mention long term implications arising from the relevant changes that are not realizable with testing frameworks alone).
We may decide that it is sufficient for a QA team to ensure that changes are trivially covered by the testing framework and that other tests don't fail. We may decide that there is no point in a QA team if it cannot make a more meaningful assessment than a binary test pass/fail, covered/not. It could be helpful to distinguish what is required for a minimally viable QA team that adds tangible value and what is required for an ideal high-performing QA team.