[LP-0004] Proposal for project governance

[llvm-beanz]

This PR introduces a proposal for project governance for the LLVM projects.

[tstellar]

Thank you for working on this proposal, I think it is really good. I do feel like there is a lot of uncertainty about how some of the policies will play out especially with the voting and team composition, so we should not be afraid to amend the policy as needed after it has been implemented.

The one thing I would like to see in the proposal is some guidance around how often the Area Teams and the Council should meet. It seems like monthly meetings would be a good idea especially since the Area Teams are responsible for ongoing maintenance of the code owner lists.

[jrtc27]

"would need to contribute code" seems too strong given we explicitly call out things like issue triaging and code review as meaningful contributions for being a voting contributor?

[tstellar]

We could allow people who don't qualify automatically to apply for voting access to a particular team and then have that team vote on it. That would help relieve some of the burden of trying to come up with a perfect metric.

What other metrics could we look at besides number of commits that are easy to quantify?

[jrtc27]

I don't know what the metric should be, I'm just saying the document isn't self-consistent at the moment, since we reuse the term "meaningful contribution" here but then go on to say it's only code contributions

[jrtc27]

e.g. if someone's code reviews for a specific backend automatically qualify them for project-wide votes then they should also automatically qualify them for LLVM Backend votes

[rengolin]

This seems to fly in the face of "We value and recognize the many shapes that contributions take."

[tstellar]

I'm starting to think that Voting by Area will be too complicated. It means we need to maintain 12 separate voting lists, 12 separate voting criteria, etc. How big of a problem would it be if everyone was eligible to vote for every area team?

[llvm-beanz]

This seems to fly in the face of "We value and recognize the many shapes that contributions take."

I do want to acknowledge this comment. You're totally right. I'm not happy with this. I've struggled to find good ways to support voting by area that didn't rely on commit filtering.

I'm starting to think that Voting by Area will be too complicated. It means we need to maintain 12 separate voting lists, 12 separate voting criteria, etc. How big of a problem would it be if everyone was eligible to vote for every area team?

Maybe this is the answer. I didn't originally have voting by area. Maybe the right solution here is to make it clear that it is okay for people to abstain from voting in areas that they're not familiar with. I personally can't imagine voting for Flang or MLIR's representation since I know so few of the people working in those areas.

[rengolin]

Yup. To complement my point on the other thread, artificial limitations are always biased and disproportionately affect certain sub-groups. However, natural limitations are much better, especially when we assume that the people vote when they have a stake or knowledge on the area, not because they're in some sort of a list. We'll get less votes, but more relevant percentage of votes.

And if we notice people trying to put out irrelevant votes just for the sake of being annoying, the CoC can deal with that already.

[vgvassilev]

IIUC, the plan is to mostly exercise voting when electing various members/community representatives. Do we plan to use such voting mechanisms to resolve disputes/disagreements?

[llvm-beanz]

No, the intent is not to enable direct democratic decision making.

[jrtc27]

FYI, CIVS has been used in the past by a member of the open-source community to send abusive polls to other open-source members (see rms-support-letter/rms-support-letter.github.io#6450 for example, though much of what I know is from private discussions). When CIVS was contacted about the abuse, the response was to filter the emails yourself, and so there is a good chance some members of our community are doing so and may not get notifications from this system. https://elekto.dev seems to be an example of a more modern site that's not so open for abuse, but there may be others, and I have no idea as to the quality of that one.

[llvm-beanz]

That is great to know. I have no experience with CIVS, I just saw that other open source communities used it. Looking at electo.dev, it looks neat, but relies on OAuth rather than email, which I'm fine with, but it has impact on how we handle voter registration and communications.

I'm going to add a note to the document that CIVS is a placeholder and we need to do a more through evaluation of our options.

[tstellar]

I think if we are going to look at a different voting tool, we should consider also switching the voting method to range voting. In my opinion the range voting process and results are much easier to understand and if you like two or more candidates the same amount then you aren't forced to place them in a random order, you can give the both the same score. Especially for elections with multiple winners, I think range voting makes the most sense.

[arsenm]

Should mention if this is a secret ballot or not

[MaskRay]

LLVM has a lot of stuff that cannot be categorized into backend or middle-end: ADT/Support, IR, testing tools, just to name a few.
My personal feeling is that LLVM components are more diversed and why selecting a code owner is extremely challenging. .github/new-prs-labeler.yml may demonstrate how diverse llvm/ is.

I understand that it can be unwieldy to have too many LLVM area teams. A thought is to add a LLVM Others or something like that.

[nhaehnle]

I second in particular the point that it should be made very clear which area the evolution of IR itself falls into, and I second the notion that it probably make sense to put it together with some of the other examples inside a dedicated area, though "Others" sounds a bit strange to my ears. Maybe "LLVM Base" or "LLVM Core" or something like that?

[pogo59]

The tools should mostly be in Backend (llc, llvm-objdump) but some obviously belong in Middle-end (opt). Although as these are frequently just drivers for library code, perhaps lumping the tools into Backend is okay.

[rengolin]

LLVM core libraries are used in Clang, MLIR, etc. That's an area that needs input from people from all projects, not just a local team. It'll be a struggle to find only 5 people here...

Back-ends can be mostly about their own communities. But if there's one area to represent all back-ends, which of them we skip when choosing 5 people for that team?

Optimization/Analysis passes and pipelines are what I'd call "Middle End", and would involve people from a lot of different backgrounds. Having 5 people to represent this whole thing would also be slightly political.

[llvm-beanz]

There are definitely flaws in the metrics I used to divide the teams. Based on the total number of unique contributors to LLVM (945), I felt comfortable splitting it in two, but not three. I was trying to keep clang (706) and clang-tools-extra (159) as two teams by slicing a bit of clang into clang-tools-extra. There is a huge flaw in my scripts that the unique contributors to clang and clang-tools-extra isn't 865 because my script doesn't account for the overlap in contributors (which I suspect is significant). That flaw also applies to LLVM's sub-directories.

I'm going to play around with making my scripts a bit smarter to see if I can get some better ideas how to split things by activity. For LLVM & Clang, where the projects are getting split, I think I'll need to be able to come up with an explicit filtering for which directories to include and exclude (if only our community had grown in the shape of our source repo 😜).

[rengolin]

I would not try to separate per number of people, but per skillset. I don't see a problem in an area team for 40 people in one side and 400 in the other...

[llvm-beanz]

I'm very open for alternative suggestions on the breakdown of area teams.

[MaskRay]

truncated content? Also 3 backquotes

[DavidSpickett]

When removed the trailing \ I got:

~/llvm-project/lldb$ python3 /tmp/foo.py
/home/david.spickett/llvm-project/lldb/include: 53
/home/david.spickett/llvm-project/lldb/source: 121
/home/david.spickett/llvm-project/lldb/include/lldb: 53
/home/david.spickett/llvm-project/lldb/source/Plugins: 103
/home/david.spickett/llvm-project/lldb:
  Alex Langford: 174
  Jonas Devlieghere: 159
  Med Ismail Bennani: 141

So seems to work fine but maybe missing some intended output.

[llvm-beanz]

I'll fix the formatting. We were iterating on the document in Google docs and some things got messed up in the conversion to markdown. The full script is there the trailing \ seems to have just been inserted by the markdown converter.

[MaskRay]

remove ` inside fenced code blocks

[vsapsai]

Suggested change

When acting to facilitate decision making the _area team _should act as a mediator between different perspectives helping find common ground and recognizing that decisions need not be binary. The _area team_ should seek to find the best solution to the framed problem, which may not be any of the proposed alternatives. If agreement cannot be reached, the _area team_ may act as the final decision maker. In that capacity decisions of an _area team_ are considered final, but can be overruled by a 2/3 majority vote of the _project council_ or the _area team_ itself revisiting the issue.

When acting to facilitate decision making the _area team_ should act as a mediator between different perspectives helping find common ground and recognizing that decisions need not be binary. The _area team_ should seek to find the best solution to the framed problem, which may not be any of the proposed alternatives. If agreement cannot be reached, the _area team_ may act as the final decision maker. In that capacity decisions of an _area team_ are considered final, but can be overruled by a 2/3 majority vote of the _project council_ or the _area team_ itself revisiting the issue.

[perlfu]

Is the intention here to imply that results are published immediately following 2 weeks of voting?
January can have either 4 or 5 Mondays, so some years this allows for a 1 week gap.

I think it should probably be considered that, in some years, the placement of this voting period can overlap significantly with extended holiday periods in Asia. Which could foreseeably produce some engagement bias.

[llvm-beanz]

Timing elections is tricky. My hope is that having a long enough voting period, and enough announcements before and during gives sufficient time for people to account for it. I'm also not at all opposed to moving the elections from January/February to a different time of the year.

[shafik]

Thank you for all the work put into this. I have not fully digested the proposal but I like the general direction.

[shafik]

C/C++ front-end feels like its own category but does not break down nicely based on directories. From a screening, code review and commits perspective this is clearly its own area.

Contributions here also are not clearly measured directly in the llvm repo but can involve WG21 and WG14 work including proposals, defect reports and participation in EWG, CWG meeting.

[llvm-beanz]

Clang not splitting by directory isn't entirely unique. LLVM doesn't split well by directory either. LLVM and Clang get enough contributor attention that I thought we needed more than one area team responsible for providing oversight to spread the load around. I'm open for suggestions about how to better make that split. My idea was basically "the language compiler" and "the language tools" recognizing that the tools rely on the compiler and there is some blurry overlap. This does need to be better specified.

I think we need to explicitly exclude participation outside the LLVM project. This isn't "C Language Governance" or "C-based Language Governance", it's "LLVM Project Governance".

I do agree that participation in the LLVM associated working groups matters, but I'm unsure if someone attending a working group meeting but not contributing to LLVM/Clang in any other way should garner voting rights.

[MaskRay]

Placing libc++/libc++abi/libunwind together makes a lot of sense. libc has a very different sub-community, so it can be moved into its own group?

lld/bolt and compiler-rt/openmp also very different sub-communities. The active contributors in lld/bolt do not overlap. The same applies to compiler-rt/openmp. If we elect someone from lld to be in the council, then that person will unlikely represent the interest of bolt.

Speaking of area team members, if each of lld's ELF/MachO/COFF/MinGW/wasm ports needs a team member, then there will be at least 5 members.

[nickdesaulniers]

Please make libc distinct from the rest of the runtimes.

[tru]

Thanks for bringing this proposal! It looks like a good step in the right direction. I think some things are still unclear that need to be clarified:

• Who is responsible for facilitating the voting? The consensus votes and the votes for the area teams. We need to have this spelled out. It could be the current project council. In any case, this needs to be clarified so that it won't be delayed or confused when it's time to vote.
• As part of the "infrastructure team" currently, one pretty frustrating thing is that even if we work on these things, not all of us have access to the servers, github, discord, etc. We need to ensure that you have access to the necessary changes if you are elected to the infrastructure team. This must be coordinated with the foundation since some infrastructure is in their hands.
• We need a "community team" or something similar with moderator access to discord and discourse. The current moderators have not been elected, just assigned. This can also be a way to acknowledge people doing a lot of work on the community side but not as much in code.
• It would be good if the project council was also responsible for electing the release managers. To make that very clear, it would also help to give the release managers a mandate.

These are my initial thoughts. Except for clarifying the voting facilitators, I don't think the rest of my points are any blockers in any way, and adopting this proposal would be a net benefit for the community.

[vsapsai]

Have a few comments about the wording. Feel free to check after the general high-level direction is established.

[vsapsai]

Maybe mention that area teams should elect a secretary as well. At least my impression was that this position remains fixed for the duration of the term and not rotated among area team members.

[vsapsai]

It's not entirely clear how to resolve conflicts between self-nomination and another-person-nomination if they are for different area teams (I have a guess how it can be done but it's not specified in the document).

Maybe add some wording that unsuccessful results in the previous election(s) don't affect the nomination eligibility [in subsequent ones]. Something to encourage people to participate again.

[nikic]

I don't like the idea of mandatory voting. I expect the "eligible voters" base is going to be quite diverse, ranging from people who make two or three commits a year, to people who make more than a thousand. The former kind of contributor may not be interested in voting (e.g. because they don't feel familiar enough with the community to vote for any particular person). And the former kind of contributor may become the latter kind from one year to the next (e.g. going from hobby contribution to an LLVM-centric job) and it seems weird to not allow them to vote just because they did not previously.

[jh7370]

I'm definitely against mandatory voting. I work on LLVM only during my working hours, and never check in at weekends/during time off etc. 14 days is a small enough window for elections that there will be plenty of people like me who will not be online during the election window because they'll be on vacation or similar. Indeed, I had 6 weeks off straight earlier this year in mid January, so would have missed any such election held this year. Maternity leave in the UK meanwhile is 12 months, and I'm sure there are other circumstances where people would need to have equal or greater periods of time off.

[stellaraccident]

+1 to removing mandatory voting from this iteration. The metric for being a member of the community in good standing has to be based on contributions. There are plenty of valid reasons that an individual may choose to not vote, including forms of conscientious objector stands, and feelings of psychological safety.

In practice, it seems like a really bad idea to roll out something as sweeping as this policy and have mandatory voting on day one. Increasing voting engagement seems more like a year three problem. Better to use carrots to get something like this off the ground vs sticks. We don't want to find ourselves in a situation a few years from now where the only members eligible to vote are the ones who are naturally biased to engage heavily in processes like this. Let that grow organically and bias towards inclusion.

[llvm-beanz]

Will remove. This was a late addition to the draft based on earlier feedback. I personally like the idea as a way to encourage contribution, but can understand the concerns.

[rengolin]

Coming from a country that has mandatory votes, this brings me many bad memories from the dictatorship era. I would not feel comfortable if this remained in the text. Thanks for removing it.

[nikic]

I have two high-level pieces of feedback:

First, while this outlines a lot of bureaucracy around how area teams are formed and fit into the larger structure, I find the proposal somewhat light on what the area teams actually do and what processes they use for that purpose. The proposal has some high level goals like "facilitate decision making", but little in the way of specifics. Does the area team only get involved in RFCs once they become contentious, or do they also have some role in the normal RFC process? E.g. does the area team have to explicitly say an RFC is approved, even if the discussion was non-contentious in the first place?

Second, I think there is an incoherence here between the required qualifications for the area teams, and their stated authority. The proposal says both that members of an area team "are not required or expected to be experts in the area the team is responsible for", but also "the area team may act as the final decision maker". I believe that either a) the area team should only be responsible for moderation and procedural matters, in which case membership can be arbitrary, or b) the area team has decision making powers, in which case it should be comprised of domain experts and have no limits on terms.

I think there is a pretty hard split in terms of overall process between these two options. With the processes outlined in this proposal, area teams should not have decision making authority on technical matters (only on procedural ones). An alternative process where they do would look much closer to how Rust teams work (where teams consist of domain experts only and new members are elected by the team itself, not a global election).

[DavidSpickett]

Should this be its own file? Advantage being you separate process changes from script changes, disadvantage being that if this script is used to implement that process, the changes won't be as strongly linked.

Though they'd be in the same repo anyway so it's not difficult to match up the process and the script versions.

[llvm-beanz]

This script is not intended to drive any official processes. I wrote it to gather metrics to help me look at contribution volume and number of active contributors in parts of the project. I do think any and all scripts we use for the election process should be published, but please don’t use my terrible Python as the basis for that.

[DavidSpickett]

Running this locally on lldb/, there is a __pycache__ folder that is a result of running the test suite. So I get fatal: Unimplemented pathspec magic '_' in ':__pycache__'.

This is not a folder git tracks, so maybe you could instruct git to ignore those somehow?

[pogo59]

The tools should mostly be in Backend (llc, llvm-objdump) but some obviously belong in Middle-end (opt). Although as these are frequently just drivers for library code, perhaps lumping the tools into Backend is okay.

[pogo59]

I don't understand why test and unittests contributions weren't counted. (They aren't real contributions? I've certainly made commits to unittests without changing anything else.)

[llvm-beanz]

As I said above, this script is not intended to drive any official processes. I just needed to have something to get a pulse of where activity is happening to drive dividing the project into teams. Any scripts used to drive real process should be less hacky.

[pogo59]

Consensus means unanimity, or close to it (close enough that any objectors don't stand in the way of a decision). Voting and surveys are ways to identify a majority opinion, which can be a long way away from consensus. Please do not confuse these concepts.

[pogo59]

Granting decision-making authority to a small group is not a consensus-seeking process. That small group might or might not implement a consensus-seeking process as it goes about making decisions.

[pogo59]

Seems like these area teams should have a chair and secretary, like all the other teams, and the chair is the liaison to the Foundation?

[pogo59]

This implies a requirement that there be at least one non-incumbent in every election slate. Who makes sure that happens?

(And who runs the elections generally? There's way too much passive voice in the whole election-process description.)

[rengolin]

I also find the restrictions of 5 and always to not be realistic over time.

We find it hard to get code owners to existing projects, and yet now we have N area teams with no overlap between them and mandate that we can always find a 6th person to replace one of the 5. This means we need (5 + 1) * N people, unless we barrel shift one person from each team onto the next, then we only really need 5 * N people, which is still a lot!

[llvm-beanz]

The current structure is 12 area teams (maybe that is too many). With that number of teams and the requirement that one candidate be non-incumbent we basically need 72 people (60 elected roles + 12 new people to cycle in).

Of the ~1700 people who contributed to LLVM in the last 12 months, I desperately hope we can find 72 who are interested in project governance.

I constantly question whether it is hard to find code owners. We have lots of active contributors who aren't code owners. Given that code owners are really only responsible for ensuring that code gets reviewed (rather than actually reviewing it), it shouldn't be that hard to fill out the owners files more robustly.

I tend to think the problem is more that people don't self-nominate and we don't have anyone responsible for nominating. People not stepping down when they move on is also a big problem because the barrier to self-nominating or nominating someone else is much higher if you're also trying to remove someone who isn't participating.

[rengolin]

Being a code owner is a thankless task. Many excellent developers with full qualifications to lead a particular area are horrible at being a code owner because they are not people persons or don't have the time to be the last line of defense.

Being a representative of an area team is less ungrateful (you actually have some tasks and recognition), but it's equally a people-person kind of task. We'll struggle to find 72 people that intersects those domains, and most likely will end up having more non-technical people in those teams than technical ones (random guessing, don't quote me on that).

On the nomination side... I'm not sure I'd like to be nominated (and then voted) in anything that I'm not comfortable. I'm not sure how that would work in practice.

[llvm-beanz]

I'll clarify the wording, but nobody nominated against their will should be placed on the ballot.

[nickdesaulniers]

So someone can only be on an area team for 2 years? That seems...aggressive. 👎

[nickdesaulniers]

This implies a requirement that there be at least one non-incumbent in every election slate.

Right. What happens when 3 people run for area team, and all 3 are incumbents? This document should clarify.

[nickdesaulniers]

Of the ~1700 people who contributed to LLVM in the last 12 months, I desperately hope we can find 72 who are interested in project governance.

Even with that number of contributors, I still suspect for certain smaller areas we will fail to fill 5 volunteered seats. Then after 2 years we will end up cutting that down due to term limits.

---

@rnk and I were discussing in person that it seems helpful that the number is odd, so that the area team can't come to a stalemate in agreement. Perhaps it's better to codify "an even number no greater than 5" than just "5" and mention this more explicitly in the doc.

[ldionne]

So someone can only be on an area team for 2 years? That seems...aggressive. 👎

This is something I find a bit icky as well -- I think that in many cases, the set of people best placed to make a decision about an area of the code doesn't change all that much over the years, and this may force us to artificially select new members of the area team that are less suited to the task. At the same time, area teams are not dictators -- they are mediators in case the normal process fails to work, so I think this makes this a bit less relevant.

Personally, I'd be happy to drop the requirement/recommendation of turnover on area teams so that in case an area team is working well there is no need to replace members just for the sake of it.

[tru]

I agree with Nick and Louis here, I think we should drop the turnover, I think it's going to be hard to fill the spots anyway.

[pogo59]

I should add that I think more structured governance is long overdue, and I'm in favor of having some people take on more proactive roles in making things work smoothly. I have some quibbles with how this proposal is described, but it seems like a reasonable enough first cut. I could wish for more evidence of checking in with software-engineering academic research, beyond looking at one book, but mostly that's just my own prejudices talking.

At least for the first round, I think The Foundation would have to run the nominating and election processes.

[lattner]

Random Q: is this really a matter for the decision making process? It is suposed to be used when there is a dispute of some sort - have you considered just starting this as an RFC to shake out the contentious issues? I'm sure there will be some ;-)

[AaronBallman]

Random Q: is this really a matter for the decision making process? It is suposed to be used when there is a dispute of some sort - have you considered just starting this as an RFC to shake out the contentious issues? I'm sure there will be some ;-)

I think this is a matter for the decision-making process in general and not just dispute resolution.

[JDevlieghere]

How were these area teams defined? For example, why does Flang have its own area time, while LLDB is part of Binary tools, despite the two projects having roughly the same number of commits over the last year?

[llvm-beanz]

I struggled with Flang because it just didn't fit with anything else. Very much open to suggestions.

[ornata]

Intuitively to me, LLDB should be its own thing considering it's a big project with a lot of moving parts that doesn't have a ton in common with other LLVM tools. It's an interactive environment that allows programmers to infer information about programs. The other tools listed consume and produce binaries.

IIUC we want to technologies (in-part) based off the amount of technical overlap contributors to those technologies probably have. E.g., people working on "runtimey stuff" probably have some sort of base "common-sense" knowledge about "runtimey things" that folks working on like, I dunno, FileCheck, aren't guaranteed to have. I can't see a ton of overlap between, say, BOLT and LLDB right now. (LLD and BOLT being together makes sense though IMO.)

So, therefore, I think that if we're going to create an umbrella for LLDB, I think it should be something like "debugging tools" or "interactive tools." Or, we don't make an umbrella and we just say LLDB is its own thing.

[JDevlieghere]

Who holds the area teams accountable?

[llvm-beanz]

The area teams are elected, so they are accountable to the community much as any elected official in government would be. They're also accountable to the CoC.

In an earlier draft I had a "no confidence" voting process to remove area team members. I removed that based on feedback that it added complication and it wasn't really clear we needed it. Terms are short, so just voting people out seems like a tractable solution.

[JDevlieghere]

Nit: the second sentence can use some commas. I had to reread it twice to understand its meaning.

[JDevlieghere]

If I read this correctly, the representative of the area team has a veto against modifying or dissolving this. I can see how that makes sense if the area team wants to voluntarily modify or dissolve itself. What kind of power does the Project Council have in the involuntary case? Is the idea that if an area team is not holding itself to its mandate, everything is resolved through the CoC? I asked earlier about who's keeping the area teams accountable, and with the powers outlined here, it doesn't appear to be the project council.

[llvm-beanz]

What about adding something like:

If the project council is seeking to dissolve an area team and the chair of that team does not consent, the team may be dissolved with a unanimous vote of the remaining project council members only after consultation with the Code of Conduct Committee to ensure that all project policies are appropriately followed.

[rengolin]

Is there a formal definition of participation with a scope as vague as "We value and recognize the many shapes that contributions take." Unless I missed something, the script below only lists code committers.

[llvm-beanz]

The script below was not intended to drive voter eligibility, it was just something I wrote to help gather some metrics to help divide area teams.

I'm looking into approaches for generating more wholistic community metrics that we can use. I won't have any updates on this for my first batch of doc updates, but I'm going to keep working at it.

[rengolin]

My worry is that any approach will have downsides and will "hurt" some people. Say the vote happens one day after 365-day period of my commit, or I have committed on a file that isn't generally attributed to an area, but the end result is, or I have organised and collaboratively wrote a Google document with 30 other developers which changed the strategy of some area for the next 5-year period, but haven't committed anything in the past year. No script will get all of those caveats.

Also in the other direction. Say I only moderate the forums, but have no idea on code quality. Does my vote on an implementation detail counts as much as the actual code owner (that commits every day)?

I'm not trying to dinimish non-code contribution, but some types of non-code contribution make it impossible to know if I'm actually capable of making an informed decision or am just giving a random opinion on something I'm not qualified (but believe I am with good intentions).

Voting is hard. Restricting voter pool is much harder and even with the best intentions is VERY likely to lead to biased exclusion, and generally the minority gets it harder. A lot of genuine restrictions (like requiring a valid ID) have unintended side-effects that can literally decide the outcome of the election.

The easiest way out is to have no restrictions and pay the cost of inaccurate results. At least they're unbiased inaccurate results. Every user associated with the LLVM org gets a vote. Non-code contributors are required to register if they want to vote. Not a deal breaker for someone who really wants to vote. Simple, effective, unbiasedly innacurate.

[rengolin]

I find it odd that area team members are not required to be experts in the area, but can unilaterally nominate code owners (who are required). I'd expect code ownership and area teams to have an intersection of at least one person. Even if nominate means putting up to a vote, if the area team is entirely non-expert, the vote isn't as meaningful.

[llvm-beanz]

Playing devil's advocate: Who says code owners are required to be technical experts?

The developer policy doesn't state that, and based on the role description in the developer policy it doesn't even seem like that requirement should be expected.

This document doesn't intend to change code owners roles or process other than having someone responsible for curating the lists. Just as today when someone "nominates" a code owner by posting on Discourse, I see the area teams as having the role of posting out that they want to nominate code owners and seeing if the community objects. No voting required.

[rengolin]

based on the role description in the developer policy it doesn't even seem like that requirement should be expected.

I strongly disagree with that statement. It may not be in the writing, but back when we were writing the document, I believe no one thought a code owner would not be a technical expert on the area they are code owners. I may be wrong, and would love to be proven so, but until we actualy find that this was the case, I would not assume so.

This document makes that assumption and encodes into writing, which is not correct. We need to make sure this is the actual consensus before encoding it in writing.

[llvm-beanz]

I think you just made my point for me. The developer policy does not place any requirement on the role, but in practice people tend to be at least technically strong in the area.

I can think of at least one instance that I was directly impacted by where a self-appointed code owner was not particularly familiar with the area they owned, and they didn't do a particularly great job of being a code owner either. At one point I even confronted them about it and got an unsatisfactory response. The problem as I see it wasn't that they were not an expert, but was rather that they didn't do anything to help get patches reviewed.

This document makes that assumption and encodes into writing, which is not correct. We need to make sure this is the actual consensus before encoding it in writing.

The document only encodes the statement as it applies to the area teams. I don't make any change to the policy for code owners. Code owners can be experts (or not) on a case-by-case basis subject to the same process (or lack thereof) that we've always had. If the Developer Policy needs to be revised to codify requirements or an election process for code owners I see that as a separate problem.

[rengolin]

The document only encodes the statement as it applies to the area teams. I don't make any change to the policy for code owners.

Yes, however, the document states: "They can nominate any individual they deem appropriate as code owner of any area they are responsible for."

What does "nominate" means here?

Today, anyone can nominate anyone to be a code owner, usually themselves, and if there's consensus, they become the code owner. But to become a code owner, they need consensus.

If area teams have the same "rights", then the text is redundant and unnecessary. Being there, could be interpreted as "more rights" and is confusing at best.

If they actually have "more rights", for instance, they can nominate and persist the change without consensus, then this is basically the same as saying they can add/change/remove code owners at will. That is not the right level of power I'd be comfortable with for an area team.

My understanding is that area teams have administrative oversight, not ultimate power. They work within the framework to resolve issues, not around it, not behind it. The way it's written gives them unfettered power to change code owners at whim.

Please understand, I'm not assuming malice here. But bias and good intentions are the source wider structural changes than malice, because it goes undetected and the direct effects are usually interpreted as "side effects".

Every time we encode something in a document, we're opening new possibilities that go far beyond what we intended, so we need to be extra careful on what and how we write.

Like a compiler pass, we should only allow what has been proven right, not what we think "might work for some cases".

[rengolin]

An individual cannot serve on two area teams

What is the rationale for this restriction?

[llvm-beanz]

The goal here is to have a diverse leadership body. Having one person sitting on multiple area teams reduces opportunities for other people in the community and doesn't necessarily help the health of the community.

[rengolin]

In my personal view, the health of the community is as important as the health of the project, not more.

In some cases, there may not be another person that can represent a project, and we'd be forced to put some random person on the team because of this restriction.

I'd make it undesirable, not forbidden.

[vgvassilev]

Some distinct llvm components have few contributors and many more contributors who are either one-time contributors or do not want to get involved more than submitting a bug report or a patch. Do you expect most of the project areas to be able to elect and maintain healthy area teams over time?

[llvm-beanz]

Some distinct llvm components have few contributors and many more contributors who are either one-time contributors or do not want to get involved more than submitting a bug report or a patch. Do you expect most of the project areas to be able to elect and maintain healthy area teams over time?

The goal in defining the areas is that they already have enough active contributors to do this today. This is part of why the metrics I used for defining the areas wasn't just the number of contributors, but also tried to take into account the number of "highly active" contributors. There wasn't a rigorous methodology used, so it is a bit hand-wavy.

[rengolin]

I also find the restrictions of 5 and always to not be realistic over time.

We find it hard to get code owners to existing projects, and yet now we have N area teams with no overlap between them and mandate that we can always find a 6th person to replace one of the 5. This means we need (5 + 1) * N people, unless we barrel shift one person from each team onto the next, then we only really need 5 * N people, which is still a lot!

[rengolin]

How do you rank contribution volume? How many commits or lines of code is worth an RFC in the forum?

In the past, we have had trouble with volume, where a small number of people were trigger happy on side projects. If this is the metric to beat, it's not unreasonable to expect that some people will (often unaware) be optimizing for that.

A more fair, but much harder, metric would be impact. I have no idea how to measure that either.

[llvm-beanz]

Dividing the project up into area teams is probably one of the most difficult parts of this proposal. Appendix 1 describes the metrics methodology I used (I'm going to add a big bolded note about how insufficient that data is, because it is insufficient).

One of my goals in splitting into area teams was to construct areas that had enough contributors that were active enough to produce area teams that cycled membership. I also looked to try and group areas of the project that had similarities. Flang is an obvious outlier where I just didn't quite know where to fit it.

I looked at a couple different bits of data, but the key metrics were: number of commits, number of authors, and commits per author. On that last point I was particularly interested in how quickly the number of commits per author fell off. For example, if you look at lld and lldb which each had similar numbers of total authors, they had very different fall off curves:

./llvm-project/lld (130 authors):
  Contributor 1: 155
  Contributor 2: 54
  Contributor 3: 29
  Contributor 4: 26
  Contributor 5: 22
  Contributor 6: 22
  Contributor 7: 17
  Contributor 8: 16
  Contributor 9: 12
  Contributor A: 11
./llvm-project/lldb (149 authors):
  Contributor 1: 174
  Contributor 2: 159
  Contributor 3: 141
  Contributor 4: 107
  Contributor 5: 101
  Contributor 6: 100
  Contributor 7: 98
  Contributor 8: 58
  Contributor 9: 52
  Contributor A: 43

My bias here was that LLDB would be likely to have more contributors who could cycle through a leadership role than LLD. This is in part because I think people who only contribute 0-2 times per month are likely less interested in being involved in project governance. I have no evidence to justify that belief other than a strong suspicion.

[rengolin]

This is in part because I think people who only contribute 0-2 times per month are likely less interested in being involved in project governance.

I don't think there's a direct correlation there. It may be reasonable to say "the probability of finding people interested is higher" but just because there's more people, not because a person who commits often is interested in project governance.

There's also many artifacts that can generate high commit rate (ex. bot CI reverts, repeated fixups, people that don't squash their PRs), which makes it a really bad metric.

Even number of PRs merged is bad. On a small sub-community, two or three developers can approve each other's patches (with good intentions and for the right reasons) much quicker than a large refactor in Clang.

Lines of code is also another bad one. I can easily beat that (unintentionally) by renaming variables or updating some tests, or adding a new benchmark.

There is no good metric that cannot be (even unintentionally) broken or abused. The best approach is to not even begin to rely on metrics.

How about we ask the people in the sub-communities how they want to be divided? @AaronBallman did an awesome job reorganizing Clang's code ownership. I would personally not have done that way (too complex), but the rest of the Clang folks seemed really happy with it, so it made me happy, too.

[rengolin]

LLVM core libraries are used in Clang, MLIR, etc. That's an area that needs input from people from all projects, not just a local team. It'll be a struggle to find only 5 people here...

Back-ends can be mostly about their own communities. But if there's one area to represent all back-ends, which of them we skip when choosing 5 people for that team?

Optimization/Analysis passes and pipelines are what I'd call "Middle End", and would involve people from a lot of different backgrounds. Having 5 people to represent this whole thing would also be slightly political.

[rengolin]

This seems to fly in the face of "We value and recognize the many shapes that contributions take."

[llvm-beanz]

@tru wrote:

• Who is responsible for facilitating the voting? The consensus votes and the votes for the area teams.

The intent here is that we shouldn't have votes for consensus. There are lots of problems with using voting to drive consensus, the intent here is to empower people to make a final decision, not describe a way to poll the community.

In my mind the LLVM Foundation can help facilitate elections, and should act as a neutral body doing so. I'll clarify that in an update.

@tru wrote:

• As part of the "infrastructure team" currently, one pretty frustrating thingis that even if we work on these things, not all of us have access to the servers, github, discord, etc. We need to ensure that you have access to the necessary changes if you are elected to the infrastructure team. This must be coordinated with the foundation since some infrastructure is in their hands.

This is somewhat of a separate issue outside this proposal. Right now administrator access to Foundation-controlled assets is pretty tightly held. The board has a line item on the agenda for our next meeting to discuss this problem.

@tru wrote:

• We need a "community team" or something similar with moderator access to discord and discourse. The current moderators have not been elected, just assigned. This can also be a way to acknowledge people doing a lot of work on the community side but not as much in code.

I think this probably touches on an area where my proposal is too vague. The elected teams in this proposal are not elected to do the work maintaining an area (although they will likely be maintainers). The purpose of the teams initially is to provide a scalable structure for decision making and delegate responsibility for maintenance of the code owners. I see this as a first step in an evolving process.

@tru wrote:

• It would be good if the project council was also responsible for electing the release managers. To make that very clear, it would also help to give the release managers a mandate.

This isn't something I had considered. I mostly tried to stay clear of redefining parts of the existing project structure (code owners, release managers, infrastructure maintainers, moderators, etc). I can see how there is probably some value here in folding the release managers in somehow to empower them and give them a clear mandate. Maybe you (@tru), @tstellar, and I can chat sometime to explore that idea.

@nikic wrote:

The proposal has some high level goals like "facilitate decision making", but little in the way of specifics. Does the area team only get involved in RFCs once they become contentious, or do they also have some role in the normal RFC process? E.g. does the area team have to explicitly say an RFC is approved, even if the discussion was non-contentious in the first place?

There is a significant amount of ambiguity here, and some of that is intentional, some of it is not. Our existing Developer Policy gives Code Owners no real authority, but as a community we implicitly grant them authority to do a lot. My goal for this policy was to grant the Area Teams a narrow slice of authority and see how the role grows.

The responsibility to "facilitate decision making" is somewhat analogous to our Code Owners having the sole responsibility to "ensure that a commit to their area of the code is appropriately reviewed". The Area teams should avoid making decisions and instead seek to help the relevant parties in the community come to agreement.

@nikic wrote:

Second, I think there is an incoherence here between the required qualifications for the area teams, and their stated authority. The proposal says both that members of an area team "are not required or expected to be experts in the area the team is responsible for", but also "the area team may act as the final decision maker". I believe that either a) the area team should only be responsible for moderation and procedural matters, in which case membership can be arbitrary, or b) the area team has decision making powers, in which case it should be comprised of domain experts and have no limits on terms.

I have two thoughts here:

1. I can't realistically imagine many people being elected to an area team with zero expertise. Our community is comprised almost exclusively of extremely intelligent technical individuals. I want the language to calibrate the bar low because in practice underrepresented minorities tend to self-select out of leadership roles if they don't feel they meet the requirements. This language is intended to head that off.
2. I probably need to clarify what it means to "act as the final decision maker". I don't think that this means pulling a decision out of thin air. This is really about resolving a disagreement with a lasting decision. The area teams should strive to act neutrally, and their decision should be based on weighing the arguments of both sides and prioritizing the long-term health of their area and community. I don't think you need to be a technical expert to do that. IMO the skills required to be good in that role aren't strictly coding/engineering skills they're soft skills.

@nikic wrote:

An alternative process where they do would look much closer to how Rust teams work (where teams consist of domain experts only and new members are elected by the team itself, not a global election).

I looked a lot at Rust. I think Rust's approach has some serious problems. The whole mess with the Rust moderation team a few years ago was deeply rooted in the "Core team" being self-appointed and not accountable to the community. The new Leadership Council they adopted shifts the problem down, but doesn't fully resolve it. The Leadership Council is elected by the teams, so they have accountability, but the teams are still self-appointed and accountable to nobody but themselves. I strived here to make each layer accountable to someone and ultimately give the community voice.

@pogo59 wrote:

I could wish for more evidence of checking in with software-engineering academic research, beyond looking at one book, but mostly that's just my own prejudices talking.

If you have any resources you would recommend I'd love to read them. I dove into this over the summer and have been leaping at everything I can get my hands on, but I don't have a lot of exposure to academic research in this area.

@pogo59 wrote:

At least for the first round, I think The Foundation would have to run the nominating and election processes.

Yea, I think the Foundaiton should take an active role in facilitating the elections. The Foundation's legal entity will be required in order to collect the voter database and comply with data privacy laws so I think that's the right approach.

@lattner wrote:

Random Q: is this really a matter for the decision making process? It is suposed to be used when there is a dispute of some sort - have you considered just starting this as an RFC to shake out the contentious issues? I'm sure there will be some ;-)

I went this direction for a couple reasons. First, I felt that we probably needed the finality that the Decision Making process provides because we're almost certainly going to settle on some details that not everyone will agree with. I also think that part of this PR will ultimately need to include a revision to the Decision Making Process document to codify responsibilities of the new teams. That's just one more reason it will need to end up here eventually.

I also felt that responding to feedback and iterating on this document in a code review rather than a discussion thread was going to be much more managable. As it is, catching up on all the feedback as I've been digging out from the Dev Meeting has been hard, but GitHub is much easier to manage than Discourse for this.

[tstellar]

What are the next steps for this proposal? It seems like there is consensus on the general idea of project governance, but maybe not on all the specifics. Should we start moving this proposal through the existing decision making process?

[llvm-beanz]

@tstellar I have some updates to the doc locally that I will try to get posted either today or Monday. Then hopefully interested parties can review and we’ll see where that lands us.

[vgvassilev]

Thank you for doing all of this work!

I have added several comments. While I understand that the intent of the document is to improve the status quo, I'd love to see a section on limitations of the current approach. That is, what would be the common pitfalls if we adopted this document. For example, what happens if the community is inactive and basically says we care more about the code and less so about voting - that would introduce challenges in representation; what are the implications of company-dominated voting; how to balance vox populi vs technical expertise; how to evolve things if this document is introduced and we need to change it; or how we can count institutional commitment - is 2 people for 5 years the same as 10 people for 1 year.

I warmly support the direction where this is going and I think our community should make sure we don't lose some of the important aspects which made us and LLVM such a great thing.

[vgvassilev]

Do we have a definition of contentious decisions?

[vgvassilev]

If we want to formalize that process of people "assigned" to mediate disputes I'd expect some sort of training for these people next to some community-agreed guidelines. Is that planned?

[llvm-beanz]

I think training is a great idea, but I don't know of any free training materials that fit what we need here nor do I think we have a good way to mandate training.

[vgvassilev]

Companies usually contribute a lot of the senior expertise in the community and they often pursue a company-endorsed agenda. In many cases voters would be exposed to particular company use-cases which would determine their voting direction.

Do we intend to get unbiased individual votes with broad use-case representation and if so is there some wording about it?

[llvm-beanz]

I don't think this is a solvable problem, nor do I think we should try to. Each person coming to the community whether they are employed by a company, a hobbyist, or from academia are going to be biased by their own goals with the project. Singling out that corporate contributors may have company-endorsed agendas isn't really helpful because a hobbyist would have their own agenda, and an academic theirs. What makes a company-endorsed agenda inherently worse?

Given that a large percentage of our community is paid contributors and they account for a huge swath of the end users of LLVM it seems that limiting the voice of corporate contributors would be detrimental to the project and community.

In a separate conversation someone commented about the possibility of a single corporate entity manipulating voting by staffing up a large number of eligible voters, and IMO we should welcome that. If a corporate contributor wants to staff 500 engineers to work on LLVM so they get a bigger voice in the community I don't see how that harms LLVM in the long run.

This all biased by my own experience of being a corporate contributor to LLVM. I've definitely had times when I felt that my employer's will for the project wasn't in the best interests of the project or community, but that has never stopped me from putting the community first.

[vgvassilev]

Please consider the use of corporate/company in terms of a big organization of people having a common goal, funding and many contributors pursuing it.

If a corporate contributor wants to staff 500 engineers to work on LLVM so they get a bigger voice in the community I don't see how that harms LLVM in the long run.

I respectfully disagree with this and LLVM has had rough rides in such cases. The community should probably consider that in some way. These 500 engineer could easily vote out diverse use-cases that enable usage in other contexts.

This all biased by my own experience of being a corporate contributor to LLVM. I've definitely had times when I felt that my employer's will for the project wasn't in the best interests of the project or community, but that has never stopped me from putting the community first.

I guess that was the motivation for my comment as some people might prioritize this differently.

[vgvassilev]

Have you considered diversifying using the a commiter per use-case/affiliation?

[llvm-beanz]

I did not consider that. I think limiting voters by affiliation would be hostile to organizational involvement in LLVM which goes against the core project principle that LLVM be friendly to corporate contributors.

[vgvassilev]

My comment was more in ensuring representation to everybody rather than anything else. Could you elaborate "hostile" in what sense?

[llvm-beanz]

This is very similar to your other comment about "balancing corporate votes". If you take away voting privileges from individuals because they work for a corporation you're adopting a policy that is hostile to corporations. Corporations are less likely to participate if they know their voices mean less proportionately.

Given how many large corporations provide massive development efforts to LLVM I think it would be an extreme disservice to the project to adopt a policy that curbed their influence because it would encourage them to fork LLVM or go their own routes instead of participating in the community.

[vgvassilev]

I am not a native speaker, please do not pick on my poor choice of the word "corporate" here. I am confused how my comment has been understood. Nobody is suggesting taking away anything from anybody. Quite the opposite ensuring that the decision making body is diversified offering inclusive environment.

Perhaps I can illustrate that with an example. It would probably be easy for a malicious actor to get funding and hire N (sufficiently large number) developers (maybe in a low income country) to develop a new LLVM target, for example. By sheer capacity this entity will dominate the backend and if wished to block development in that area. That example is hypothetical but quite possible in my point of view.

I am trying to outline the need to protect ourselves from such situations to the aid of everybody including the large entities which may not be the largest. There are multiple comments in that vain to diversify by skillset, etc, which perhaps goes in a similar direction.

The rules of the community as proposed here should encourage respecting the diversity of LLVM use cases no matter who pays most today. The next tricky question is how can you evaluate what would be a massive development effort? Is that 10 people put by a private entity for 2 year (that can disappear in the first quarter of year 1) or 2 people put by a national lab for 10 years (that are likely to stay beyond that)?

Different entities with different funding streams have different pros and cons -- giving most power to the strongest of the day does not sounds like an optimal solution.

[pogo59]

Different entities with different funding streams have different pros and cons -- giving most power to the strongest of the day does not sounds like an optimal solution.

I think there is no optimal solution; there is actual research into voting systems to demonstrate that, which I read decades ago. At some point we have to go for good-enough, and if it blows up, we need to try again. I don't think any company would be motivated to take over governance of LLVM, it would be cheaper to fork it and go their own way.

[rengolin]

That example is hypothetical but quite possible in my point of view.

We have most back-ends covered today and I don't see this being a problem in any real context.

More generally, this strategy would (or should) fall foul of the code of conduct, in which case we don't need extra policies in place to protect against.

Am I being too naive?

[llvm-beanz]

I agree with @rengolin that any actively malicious behavior should be controllable via the code of conduct, and the proposal does explicitly state that the CoC applies to all governance activities.

I think anything you do to prevent a company from hiring 100 engineers to gain influence over part of the project would back-fire by diluting the influence of well-meaning companies and deter corporate participation.

[nhaehnle]

I do think the underlying concern is a legitimate one, not because of any extreme examples where a company goes out of its way to be specifically malicious. It's great if company X invests a lot into LLVM overall, but if an area team then ends up being dominated by folks from company X, it is fair to say that that is almost certainly a bad outcome -- because they will be biased. (What does "dominated" mean? I'm not sure, but I think if people from X are an absolute majority in an area team, that should raise some eyebrows.)

At its core, this is about ensuring that a diversity of viewpoints is represented and, quite frankly, at least for technically-oriented area teams, what company somebody works for is probably more relevant there on average than any other consideration.

On the other hand, I suspect that attempting to cure this with a formal mechanism is likely worse than the (risk of) disease.

We should be able to agree though that there is a potential bad outcome here and that as a matter of LLVM culture we want to avoid it.

[tstellar]

I think if we want to impose limits on corporate representation, it should be done by limiting representation on the sub-teams/steering committee rather than on the voters. I know some other projects do this. I'm personally OK if we have no limits at all on representation, but I think limiting voters is not the right way to go.

[vgvassilev]

Some distinct llvm components have few contributors and many more contributors who are either one-time contributors or do not want to get involved more than submitting a bug report or a patch. Do you expect most of the project areas to be able to elect and maintain healthy area teams over time?

[vgvassilev]

Could we call out a section explicitly on Training & Education? This has been historically undervalued area of work which practically ensures a stream of new developers. In particular, recent years many educational institutions don't have even a compiler construction course which makes raising the next generation of LLVM developers quite hard.

[vgvassilev]

IIUC, the plan is to mostly exercise voting when electing various members/community representatives. Do we plan to use such voting mechanisms to resolve disputes/disagreements?

[vgvassilev]

Can we say something about the dissolving area teams in a motivated in writing way by the project council.

[vgvassilev]

Can we add "in a motivated in writing way".

[vgvassilev]

Is there a way to balance the corporate vote? That is, in many areas one or two entities provide most of the workforce and it is quite easy to skew the process even unintentionally. I realize that's very hard issue to tackle but I see this as a critical aspect for keeping the overall community health and the diverse llvm use-cases.

[tstellar]

I still think it's going to be too difficult to determine eligibility for each area team. I would recommend that we just have a single pool of voters. Given that one member from each area team will serve on the project council, and will make decisions that affect the whole project, I think it still useful to have people vote in areas they don't actively contribute to of they want. If we are really concerned about this, maybe we could limit the number of area teams people can vote for (e.g. You can only vote in 5 area team elections). This kind of restriction is going to be much easier to implement than trying to group voters based on their contributions.

[maksfb]

Boosting the visibility for earlier comment by @MaskRay: there's close to zero overlap between LLD and BOLT.

[nickdesaulniers]

I also think about the llvm-project's substitutes for GNU binutils. llvm-objdump, llvm-objcopy, llvm-readelf, llvm-nm, etc.

[MaskRay]

I really like the proposal for improving project governance, but I also share the concerns raised about area teams in the other replies. In particular:

#54 (comment) (nikic)

... I believe that either a) the area team should only be responsible for moderation and procedural matters, in which case membership can be arbitrary, or b) the area team has decision making powers, in which case it should be comprised of domain experts and have no limits on terms.
I think there is a pretty hard split in terms of overall process between these two options. With the processes outlined in this proposal, area teams should not have decision making authority on technical matters (only on procedural ones).

#54 (comment) (rjmccall)

My second thought is that the Area Teams concept seems really strange. I don't know who's going to want to serve on all these little committees that, frankly, don't seem to do very much on their own. They explicitly don't have any technical authority, or even expertise, and seem to be primarily be conflict mediators. Are we expecting this to be a way of showing leadership and gaining credibility in the community? Why is this role even area-specific? I'm not sure I as a contributor would know who these people are, especially given the requirement that they rotate on a fairly quick basis.

@rjmccall's suggestion to "build on and broaden the existing concept of code ownership" is an interesting alternative.
Perhaps we should explore this option further before establishing a new group.

Once we have a good definition of area teams and decide to move forward, I agree with the idea of starting with a single, well-defined area like some part of llvm/ or clang/. This would allow us to test and refine the approach before scaling it up.
Some subprojects might have relatively smaller communities and fewer RFC proposals that would lead to disagreements, making a dedicated area team feel like an unnecessary burden.

[arsenm]

I think this is essentially fine. It feels like this proposal is stuck in the indecision quagmire it's intending to solve

[arsenm]

Should mention if this is a secret ballot or not

[ldionne]

When I read the original proposal, I was worried about a few things. The aspect that worried me the most was the lack of requirement of technical expertise for area team members, and I believe that concern is what scares other folks too.

However, I feel like this last revision does a good job at clarifying the intent: since area team members are elected, it's unlikely that folks who shouldn't really be on area team X will end up on it. I also found the new revision to be a lot clearer about when we intend area teams to actually take over, and I get the feeling that it should happen fairly rarely. I can think of a few debates in the past couple of years that might have ended up at the area team or project council level, and none of the ones that come up to my mind were deeply technical. All in all, I still expect the vast majority of technical decisions (even hard ones) to be resolved at the RFC level without having to involve area teams.

All in all, I think this is good enough for me, at least as a first step. The main risk I see is for this new structure to be abused to force decisions upon projects by escalating RFCs to area teams whenever the normal consensus seeking process doesn't go the way someone desires. But if this were to happen, we could always raise that issue and revisit this proposal accordingly.

So my opinion is: let's try this out, it's better than what we've got and we can improve it in the future as needed. I also think we have to be careful not to stall this proposal on "ifs" and "maybes" unless we have concrete actionable alternatives to propose - it's easy to fall into that trap but it doesn't get us anywhere.

[ldionne]

Suggested change

	Each _area team_ and the _project Council_ should have one scheduled public meeting per month. The date of the scheduled meeting should be on the LLVM Community Calendar. The calendar invite will have a link to a public meeting agenda. The teams may have non-public meetings for discussion, deliberation. planning or other purposes.
	Each _area team_ and the _project Council_ should have one scheduled public meeting per month. The date of the scheduled meeting should be on the LLVM Community Calendar. The calendar invite will have a link to a public meeting agenda. The teams may have non-public meetings for discussion, deliberation, planning or other purposes.

[ldionne]

Suggested change

There has been a perception that this proposal comes from the LLVM Foundation exerting its will on the community. That could not be further from the truth. While members of the LLVM Foundation board have driving this and deeply invested in this proposal, the proposal did not come out of the LLVM Foundation. In fact, it was iterated on and circulated with all the co-signed authors before it was shared with the LLVM Foundation Board.

There has been a perception that this proposal comes from the LLVM Foundation exerting its will on the community. That could not be further from the truth. While members of the LLVM Foundation board are driving this and deeply invested in this proposal, the proposal did not come out of the LLVM Foundation. In fact, it was iterated on and circulated with all the co-signed authors before it was shared with the LLVM Foundation Board.

Is that what you meant?

[nickdesaulniers]

My second thought is that the Area Teams concept seems really strange.
It seems to me like a better approach would be to build on and broaden the existing concept of code ownership.

+1

I could see if the Area Team was self-forming, but having other folks who are llvm contributors but maybe not contributors to that sub project vote seems like a way for uninformed voters to choose project governance for Areas which they know little.

Let's say a subproject today has a clear and active code owner. This area team proposal sounds like it will take decision making power away from the code owner, and give it to a group of folks who were voted on by people who mostly don't contribute to their area. I'm not convinced that just because someone has contributed to llvm that they know best who should serve as the decision makers for a subproject/area.

Requiring 2 year term limits (for the area teams) also seems aggressive and sounds like it will result in churn in the community. Imagine being very active in project governance, then being told you can't make decisions anymore because you've been active for too long. I bet folks will stop contributing over that. We want to encourage people to be active for longer, not less!

Having the project committee arbitrate when an area or code owner can't come to a decision seems useful. To me, it's more important that the project committee ensure the health of the llvm community by ensuring the area teams (or code owners) are active, and if not, doing something about that. Having a code owner that's inactive is the largest deficit in the LLVM community IMO.

b) the area team has decision making powers, in which case it should be comprised of domain experts and have no limits on terms.

+1

[nickdesaulniers]

s/email threads/LLVM Discourse threads/

[nickdesaulniers]

s/the LLVM Community Code of Conduct/the LLVM Community Code of Conduct/

s/Code of Conduct Committee/Code of Conduct Committee/

(add links)

[nickdesaulniers]

The second sentence here is a repeat of L81. Consider dropping it.

[nickdesaulniers]

What happens when someone resigns and the area team doesn't appoint a replacement?

---

Can we have some kind of stronger guarantees that folks resigning ensure they appoint a replacement? Otherwise perhaps the area team should be dissolved by the project council.

[nickdesaulniers]

the area team may act as the final decision maker.

What if the area team is in disagreement? How does that get resolved?

---

Seems like the project council gets involved. Perhaps worth alluding to that earlier here as well.

[nickdesaulniers]

So someone can only be on an area team for 2 years? That seems...aggressive. 👎

[nickdesaulniers]

Please make libc distinct from the rest of the runtimes.

[nickdesaulniers]

I also think about the llvm-project's substitutes for GNU binutils. llvm-objdump, llvm-objcopy, llvm-readelf, llvm-nm, etc.

[tru]

As many others had said, I think this is a very good first attempt. There will probably be things that we need to tweak and change down the line, but I think we need to improve on the current status quo and I think this is a great first step. My vote is to implement this now and treat it as anything else in the code base and iterate/improve when it's necessary.

[AaronBallman]

My second thought is that the Area Teams concept seems really strange.
It seems to me like a better approach would be to build on and broaden the existing concept of code ownership.

+1

I have some perspective on this in terms of Clang code ownership. The feedback I've gotten from Clang code owners can be summarized as "I am drowning with community responsibilities already, please do not give me more responsibilities." In fact, I would not be surprised if making area team membership a mandatory aspect of code ownership led to (a small number of) code owners stepping back despite otherwise being active in their role.

I think it's perfectly reasonable to encourage code owners to join an area team, but I'd be wary of tying the two responsibilities together directly.

Also, another thing I think is important to consider is that code ownership is largely about managing technology while area team membership is largely about managing people via a process. Code ownership still touches on people things and area team membership still touches on technical things, but I think the skill set needed to be successful in the roles might be slightly different.

[stellaraccident]

My second thought is that the Area Teams concept seems really strange.
It seems to me like a better approach would be to build on and broaden the existing concept of code ownership.

+1

I have some perspective on this in terms of Clang code ownership. The feedback I've gotten from Clang code owners can be summarized as "I am drowning with community responsibilities already, please do not give me more responsibilities." In fact, I would not be surprised if making area team membership a mandatory aspect of code ownership led to (a small number of) code owners stepping back despite otherwise being active in their role.

I think it's perfectly reasonable to encourage code owners to join an area team, but I'd be wary of tying the two responsibilities together directly.

Also, another thing I think is important to consider is that code ownership is largely about managing technology while area team membership is largely about managing people via a process. Code ownership still touches on people things and area team membership still touches on technical things, but I think the skill set needed to be successful in the roles might be slightly different.

+1 - I am also struggling with the commingling of technical and administrative/management responsibility and reconciling it with the fact that many of the most productive contributors will see this as, at best, a necessary service vs something sought after. I'm struggling with how to phrase this without coming off as rude, but the thing I want for the project is for the most level headed pragmatists who don't necessarily want the job to be the ones that make it to these key positions that have any implied authority. In effect, I want the people who are conscientious, mostly keeping their heads down, and have a broadly developed view of what is going on to be empowered. I'd also like for them to not be drowned in administrative duties.

I don't know how to reconcile those desires, but what I fear is that we create something that starts to feel more like a home owners association vs an act of service.

Perhaps instead of codifying very detailed responsibilities, term limits, voting requirements and rules up front, we put a one year time limit on the process itself and require that we draft a v2 based on learnings from the teams that were the most effective over that term. Something like that might help us grow into things while also giving all of us a right of review, so we know things can be fixed without a lot of drama. i.e. when faced with a big change and a lot of people, one can either nail everything down in a process of consensus or the cost of getting it slightly wrong can be reduced by explicitly making it a thing that we agree to rev and improve.

Just my 2 cents.

(sorry for delayed feedback - I've been mulling this since it was posted and trying to figure out how to verbalize what isn't sitting right with me. thanks to the people who started hitting approve to get me to actually write down my thoughts)

[rengolin]

Perhaps instead of codifying very detailed responsibilities, term limits, voting requirements and rules up front, we put a one year time limit on the process itself and require that we draft a v2 based on learnings from the teams that were the most effective over that term. Something like that might help us grow into things while also giving all of us a right of review, so we know things can be fixed without a lot of drama

+1. This is basically the same as some of my comments up-thread: Start much simpler, fail, iterate.

I'd start with understanding why we need such a council. Is it because code owners are not doing that job? Not supposed to do that job? Not easily removable? Can/should we fix the definition of code owners? Maybe just add more roles (admin, conflict resolution, community) and follow the current process we have for code owners?

There are many easier ways to be inclusive and to allow non-technical people to contribute to the core of our community without creating hierarchies and power division, voting schemes and fixed terms.

I always go back to @AaronBallman's reorg of Clang code ownership. That was a much more efficient process because it didn't change a lot, but it achieved a lot. Perhaps we can start from there, enhance the roles (not just "code" owner) and iterate through the sub-projects.

If some emergent feature becomes obvious, we encode it. Like we've always done in LLVM, our process is to document policy from existing behaviour, not to try and force behaviour through new policy.

FWIW, I am not ok with this current proposal for project governance. I won't stand against if there is consensus, but I don't think we're going in the right direction.

More importantly, I don't think we'll be more inclusive. On the contrary, I believe we'll enforce conscious bias through policy in an attempt to remove unconscious bias through actions.

[stellaraccident]

I'm not going to go so far as to say I don't support this being tried in some fashion. But I do believe this approach is likely setting us up so that common sense does not have the upper hand over time.

I do agree with the sentiment that something needs to change. What we are doing isn't working at the present scale. But we are not at the bottom: do this wrong and it can get worse.

I'm willing to try anything for a year. In fact when it comes to change, I prefer to be able to just put the objection out of my mind and give it a try -- taking a break from trying to logic my way to something better and giving myself time to adapt. From that perspective, I'd rather see something time limited. We can always pull the plug if it is a disaster at any time, but planning for the more likely scenario that it helps some things and regresses others, I work better knowing there is a future date we can address that stuff vs trying to incremental it in real time. Both let's me park a mental thread and gives time for my gray matter to assume a new form before spending mental energy on a prior state.

[tstellar]

I think this is a great improvement!

[rengolin]

I'm not going to go so far as to say I don't support this being tried in some fashion. But I do believe this approach is likely setting us up so that common sense does not have the upper hand over time.

I do agree with the sentiment that something needs to change. What we are doing isn't working at the present scale. But we are not at the bottom: do this wrong and it can get worse.

I've been pointed back to this review recently, and I want to make clear my words probably sound harsher than I wanted them to be. What I was trying to say is exactly what @stellaraccident said here. That's why I "thumbed up", but that doesn't help other reading my own reply.

So let me be clear:

• I am not against this proposal: I think it's long due and we really need this. I'd love for us to have something to iterate on.
• I am unhappy that we're encoding too many hypotheticals, which can serve as undue metrics that people can use (unintentionally) as arguments in upstream discussions ("the doc says so"). I'd rather use common sense, but sense is rarely common.
• I would very much prefer a leaner document that increases the number of cases over time, rather than a document we need to remove stuff because they have been weaponized. Both cause grief, but adding actual facts is easier than removing badly worded hypotheticals.
• I would strongly encourage us to have a paragraph with the following sentiment: "This document is a reflection of what we want the community to be. It may be wrongly worded, it is biased by the current writers, it will definitely get 'dated' as our community evolves. When any of those problems are detected, the first reaction must be to re-evaluate the document, not to use it as misguided direction because someone once wrote it."
• A follow up paragraph could be due: "However, we truly believe in what we write, so changing the document must be a new consensus forming process, and not cheaply done. More importantly, this is not just a description of the 'status quo' that we're trying to maintain, it's a proposal of the community that we want to be, and in that, it has powerful consequences and should not be treated lightly."

I'm willing to try anything for a year. In fact when it comes to change, I prefer to be able to just put the objection out of my mind and give it a try -- taking a break from trying to logic my way to something better and giving myself time to adapt.

+1 to this. Let's get something done and try it out, and if it doesn't work, we adapt. Nothing we haven't done before as a community.

[dwblaikie]

As a member of the review committee for this proposal, and based on a formal meeting according to the process of the proposal process, I'm approving this PR /as a draft/. It should be committed so it's easier for us to continue iterating on it - this should not be construed as any final/formal/complete approval of the proposal.

[tstellar]

Approved with the same rationale as @dwblaikie .

[ldionne]

Approving as a draft as well. Some action items have been identified and they will be addressed before the proposal is finalized, but for now merging this will make it easier to iterate.

I think I can speak for everyone involved in the proposal and its review process to thank all the folks that participated in the discussion (here, on Discord, round tables, etc) and provided feedback.

[MaskRay]

llvm/clang/lldb have switched from "code owners" to "maintainers". This might need rewording.

[ldionne]

Yes, this has been discussed and @llvm-beanz said he would update to the new wording.

[MaskRay]

Took another look. LGTM. The ~20 code owners occurrences need to update.

(Thanks for decreasing the number of area teams. Previously many area teams without many active contributors were mentioned and they likely don't need decision making anyway...)

[llvm-beanz]

Took another look. LGTM.

Thank you!

The ~20 code owners occurrences need to update.

This is on the list of updates requested after the review manager meeting. I'm going to merge the PR as-is, and post a new PR shortly updating with the feedback from the review manager meeting.