Process requirement for sufficient demand

[RossTate]

WebAssembly has a significant backwards compatibility constraint. It is also exploring an almost entirely new space and solving very hard problems requiring substantial design and implementation work. As such, it is important to make sure that any designs developed, implemented, and shipped have been substantially evaluated on whether they achieve their intended purpose. To this end, I suggest the following amendment to the process:

As a requirement for advancing to Stage 3, every feature of the proposal should have approval from a substantial user base (where the appropriate scale/meaning of "substantial" and "user base" depends on the specifics of the proposal).

There are two primary reasons for this revision:

1. We cannot evaluate whether the proposal's design for a feature effectively achieves its intended goal without a substantial user base to either demonstrate its utility or to provide feedback on how the proposal needs to be revised to meet their needs.
2. As much of that evaluation should be done prior to implementation to reduce churn due to design changes.

With this requirement in place, when the reference types proposal went to Stage 3 we would have seen that, although external references had a big user base, subtyping did not and so should be carved out into a separate proposal. Furthermore, this separation would have made it apparent that many other proposals do not need to depend on the reference types proposal. Many uses of type imports do not need any constraints at all (arguably external references are an example of this), and alternative (even more expressive) constraints besides subtyping could be explored. Although exception handling uses exnref, which might represent uninterpretable exceptions depending on the host, there is no requirement that exnref be related to external references. Even garbage collection does not depend on anyref or external references; we demonstrated that even languages relying on a uniform representation would generally prefer to use a uniform representation that is specialized to the needs of the language over a useless type like anyref. So the separation this requirement would have encouraged would have also reduced the dependencies between proposals.

But pulling back from the current situation, what do y'all think of the proposed revision that every feature of a Stage 3 proposal should have enough demand to warrant implementation and to enable proper evaluation?

[This was substantially edited thanks to feedback from @jgravelle-google.]

[aheejin]

There is no need for exception handling to depend on external references, and you can imagine many applications of WebAssembly with exception handling that would not even want external references.

Please. Your opinion is not the only one that matters. After endless back and forth in the EH repo, this kind of assertion comes across as not only too assertive but even arrogant.

[tlively]

The current requirement closest to this proposal is the stage 4 entry requirement that "At least one toolchain implements the feature." We have limited data because not many proposals have moved to phase 4, but so far it has worked out. That being said, thinking more about the needs to toolchains and source languages earlier in the process can never be a bad thing.

[RossTate]

Oh shoot, I meant to clarify that I was not asserting the removal of exnref. I was just asserting that exnref is independent of anyref. Does that address your concern, @aheejin?

[aheejin]

exnref is an external reference, even if it is not a subtype of anyref, in that it is created by the embedder, isn't it?

[RossTate]

No. An external reference is something that is not interpreted by the wasm code at all (except through callbacks to the environment via imported functions). An exnref is an internal reference, with a meaning that is interpreted and utilized internally by the wasm program. Sorry for the lack of clarity.

[RossTate]

That said, there is some mixing in the sense that you can catch exceptions thrown by JavaScript (it's just that br_on_exn will always fail). So I guess it would be more precise to say that exnref does not rely on an extref type.

[tlively]

Let's keep this on topic. This is not an issue about EH.

[kripken]

(What is the topic? When I read

The current situation we are currently dealing with is first and foremost due a shortcoming in process. We have a feature that is about to ship

I don't know what "the current situation" and "a feature" refer to...)

[RossTate]

Actually, I appreciated @aheejin's pushback, as it helped clarify my intent. But I guess your point is that we should focus on the high-level topic independent of the current situation, which makes sense.

The current requirement closest to this proposal is the stage 4 entry requirement that "At least one toolchain implements the feature."

This is certainly a good requirement to have, as toolchains are extremely important for practicality and a good test of implementability and maintability. Indeed, in some cases I expect the "user base" will be the toolchain itself. There have already been a number of times where the primary value of a feature has been in its utility for tooling.

But when toolchains or engines are not the "user base", neither this requirement nor the corresponding requirement for engines test whether the design achieves its intended purpose.

but so far it has worked out

So far you have had a major user base for every feature, namely C/C++/LLVM implementors and developers. But now that's no longer case: C/C++/LLVM isn't utilizing the proposed subtyping. I think that is what's fundamentally different here. We have a feature that is not useful for wasm's current user base, and the concern I am raising is that I have reason to believe that the design of this feature does not match the needs of the potential user base that would most rely on this feature, but because that potential user base is not at present an actual user base we currently have no way evaluating whether or not the design serves its need.

[RossTate]

@kripken The current situation I aluded to is that the reference types proposal has two primary features, external references and subtyping, one of which is there is great pressing need for and another one of which there are concerns about (primarily raised by me) and no pressing need for. This situation is complicated by the fact that there is no current user base for subtyping, so we are having a difficult time evaluating the merits of the design. So the high-level purpose of this topic is to see if we can come up with a good way to prevent bundling of high-priority features with (currently) unneeded features, and to ensure that we have a good basis for evaluating a design before it progresses so far. (Hope that made sense.)

[jgravelle-google]

Not having followed the subtyping story too closely, I don't have much of a dog in this race. With that in mind, I would like to highlight how hostile the tone of this issue comes across as. I also think you have a valid point, but it's buried under enough vitriol that it's hard to want to engage.

The current situation we are currently dealing with is first and foremost due a shortcoming in process.

That is a bold claim. To me it reads "your process is wrong, and so obviously wrong that anyone can see it."

We should not be wasting everyone's time implementing features that are not currently needed.

That is highly uncharitable. 1) "The work you are doing is a waste of time" and 2) it supposes that no justification exists for why a given proposal is important. No one is implementing proposals just for the hell of it here.

This is certainly standard process in both language design and software engineering,

"This is obvious, but I must have been the first one to actually say something".

a feature everyone wants is being held hostage by a feature no one currently needs or even properly understands the impact of.

Everyone and no one are the strongest and most sweeping of generalizations. It is difficult not to read that as hyperbolic and reactionary. Not only presumptuous to assume one can speak for everyone, not only dismissive to assume nobody has any ideas about the impact.

"held hostage", additionally, is a heavily emotion-laden phrase.

---

I could continue but you get the idea.

I would like to point out that web standards work is only possible through a great deal of social goodwill. Being kind to one another is actually a core component of the work we do. And it is real work, we have a tendency to be deeply passionate about the future of the web, and when our visions diverge the tensions can rise quite quickly. Which is why we need to be so cautious and careful about the manner in which we communicate.

[tlively]

So far you have had a major user base for every feature, namely C/C++/LLVM implementors and developers. But now that's no longer case: C/C++/LLVM isn't utilizing the proposed subtyping. I think that is what's fundamentally different here

That makes sense. Another example is in the bulk operations proposal, where the memory operations were important for threading support in the toolchain, but the bulk table operations were not immediately useful. Now the proposal is being held up by a dependency required only for those bulk table operations.

I guess my takeaway here is that we need to be mindful of minimum useful functionality not just when we create new proposals but also when we split things off of existing proposals.

[binji]

I guess my takeaway here is that we need to be mindful of minimum useful functionality not just when we create new proposals but also when we split things off of existing proposals.

I agree with this. In particular, we ask for motivations when we initially create a proposal. As we move forward with a proposal, we should evaluate whether our proposed design, with all its component parts, fulfills those motivations. Conversely, we should ask whether the components of the design have significant motivation, and if not, consider whether they should be a part of this proposal.

That said, I don't think the proposed phrasing addresses this concern. Instead, it sounds to me like it requires surveying users, or finding some other way to seek "approval".

[RossTate]

@jgravelle-google, thanks for calling me out on the hostile tone. I let my frustrations get the best of me. Hopefully it's alright that I edited the post to make for a more welcoming entrance for new participants in the conversation, and hopefully I managed to address the hostility while still presenting the case.

[RossTate]

Instead, it sounds to me like it requires surveying users, or finding some other way to seek "approval".

This was my own primary concern with my own phrasing as well. I put that together as a starting-off point, and I am definitely open to suggestions (including complete rephrasings).

[jgravelle-google]

hopefully I managed to address the hostility while still presenting the case.

Thanks. I think that reads much clearer now actually, well done.

As we move forward with a proposal, we should evaluate whether our proposed design, with all its component parts, fulfills those motivations.

It might be proposal-specific what the evaluation criteria actually are. What we could do is require some explicit validation criteria, e.g. "implemented in LLVM and V8, reduces code size by >1%", as a minimum early-validation needed to advance to stage 3. Declaring such a criteria would be required to move to stage 2, and it helps make the development plan and goals of the proposal more concrete. For example, Interface Types would want at least 2 languages able to generate adapter functions and communicate a nontrivial data type with a differing ABI. These would be necessary-but-not-sufficient conditions and should improve our confidence.

[tlively]

I like that idea. It's very similar to an idea raised multiple times as a way to make the "web VMs" requirement more flexible and I saw that a similar mechanism was also recently adopted by WASI in their own phases process.

[jgravelle-google]

To clarify, I'm not saying we should loosen the web VM wording, or anything related to stage 4. Maybe we should, but we could independently have feature-specific requirements for earlier stages too.

[RossTate]

Conversely, we should ask whether the components of the design have significant motivation, and if not, consider whether they should be a part of this proposal.

I think this emphasizes a point I meant to place more focus on. A proposal is a collection of features. Part of the process for advancing to a certain stage should include 1) an evaluation of the utility of each feature in isolation, and 2) a plan for evaluating the design of each feature in isolation. Of course, there will often be dependencies between features that should be accounted for, but part of the process should be evaluating how subcomponents of a feature might be carved out. (I mentioned in certain settings the idea that proposals might be subdivided into stages, each of which could be in different phases, and this process would help identify features that would be better pushed back to a later stage.)

(This comment is meant to be complementary to, not contradictory to, the above suggestions.)