Video recording: https://www.youtube.com/watch?v=936z41Mo1tQ
- Agree to Membership Agreement, Participation Guidelines and Code of Conduct (1m, Lee)
- Introduction of attendees (5m, Lee)
- Determine volunteers for note taking (1m, Lee)
- Review agenda (2m, Lee)
- Review previous meeting's action items (5m, Lee)
- GraphQL Scalars hosting and coordination (10m, Evan and Andi)
- GraphQL namespaces (20m, Courtney)
- Input Union Criteria: Unconstrained combination of input types to unions (20m, Benedikt)
- Input Union & Tagged Types (30m, Lee, Benjie, Evan, Vince)
- ADD YOUR AGENDA ABOVE
- Benjie
- Stephen
- Alan
- All action items
- Lee: in future we'll try to create action items on the day after rather than before the meeting ;)
- Makes it easy to see if actions are open or closed
- Lee: Making sure we have agenda items for the rest of the year ✅
- Introspection shortcuts RFC: discussed as potential Stage 1; decided that it needs a clear problem definition; still open
- Update Input Union RFCs: we've had two meetings but haven't captured all of the learning into that doc yet; anyone on the subcommitted can do this
- Add InputUnion solution criteria that error messages are well specified - Pending
- Clear problem statements for @defer/@stream: ✅
- Open action items remain, but hopefully having the issues to track them will help to make sure we don't lose track of them
- Benedikt: how do I join the Input Unions subcommittee?
- Lee: it's not really a subcommittee, more a meeting between the champions of the various proposals - Benjie, Evan, Vince and Lee.
- Ivan: I'll be opening a draft RFC shortly after the meeting, it's almost ready.
- graphql-scalars.com
- Evan: Since last time we shipped the graphql scalar spec change (all done now) Andi created a website and hosted a simple DateTime spec (the original motivator). We had some brief convo where this should live.
- Question: Where should this live? Is it official? It is currently just Andi’s personal website.
- Lee: Most important question is if this graphql-scalar website has a high barrier to entry for all scalars. Or is it just supposed to be a centralized hub with low barrier to entry?
- Ivan: It is like the chicken/egg problem.
- If we start using the DateTime scalar in GraphQL-js then it'll help push the popularity of the site, but I'm not keen to use a third-party URL - it's a question of whether it's reliable or not - what happens if the URL disappears? GraphQL Cats is a similar example.
- Courtney: I concur with the opinion. I suggest that the website should be hosted in a GitHub repository. It would give us more confidence if it’s in a repo. It will also take away from the potential fragmentation and reduce duplication with new scalar proposals.
- Evan: it's a GitHub Pages website, so it's already a repository 👍
- Lee: the reason I'm asking this is because how we propose this to people is important. It seems more valuable for it to be "centralized" than for it to be "blessed" (maybe not right word). Not a guarantee that you won't have issues with it, but like an npm-style repository of all the specs.
- Benedikt: We will need to think about naming. It would be undesirable if bad specs took up all the best names.
- Lee: name collision; are pages mutable? What if clauses are missing? in npm you'd introduce a version breaking change; but we've not baked semver or anything like that into the custom scalars spec, but you may want to have a way of versioning so that you know a specific version is immutable.
- Evan: Reminds me of golang packaging (they are by URL)
- Lee: How does that work?
- Evan: In practice, people use proxies, which also contains the version
- (example in chat) https://gopkg.in/Shopify/sarama.v1 provides a proxy to https://github.com/Shopify/sarama
- Ivan: I think we can use sequential numbers like RFC, this allows for small changes, it's not like names, people are okay with standards having numbers. I researched the RFC process; there are types of RFC, for informal they just check your spelling/grammar/formatting/etc and publish it without analysing the content. We could have these types of RFC specified in the URL e.g. scalars/experimental/RFC1234
- Ivan: sequential numbers are better than a hash.
- Lee: I agree, sequential numbers are easier to understand.
- Lee: the URL proxy pattern is interesting, looks like a good way to handle the problem.
- Benedikt: Deno does the same thing https://deno.land/x
- Evan: it has some logic around git tags and what not, for semver-like branches.
- Lee: What do we need to decide on?
- Evan: it seems that there's an appetite to put this under the GraphQL Foundation umbrella, but I'm not sure if anyone has the time and interest to chase this up - someone needs the time and access to do the GraphQL Foundation side of things.
- Ivan: We had a discussion with Andi and it seems that he is open to new ideas, however, he doesn’t want to spend too much time on reviewing and procedures. I cannot speak for him...
- Lee: Ivan, I agree with you that there's an appetite for it to be a Foundation project to ensure it has a long life, but each individual document should probably default to this "informal" style. The next right piece might be to remove even more hurdles - do you need to get a PR merged? How do you get something in there? What happens when people disagree about a scalar; can we allow that and have a way to make it clear? npm seems to be encouraging using "scopes" (@foo/packageName) - do we do this as for example graphql-scalars.com/@andi/RFC1 ?
- Benedikt: we don't need to build a new service like npm around this; we should take advantage of GitHub as an ecosystem and leverage the code owners files; sub-directories for each scalar, and code owners can do whatever they want in each branch. We can disable force pushes so things remain immutable.
- Evan: who's going to set this up and maintain it? Seems Andi doesn't want to, and I don't have time.
- Evan: two questions: survivability(?) - we want to make sure things don't go away if Andi stops renewing the domain bill. We can fix this by transferring the domain to the Foundation. Question two is who is going to do the work?
- Gabriel: For the most part, the GraphQL Foundation is made up of volunteers.
- Courtney: if we define a clear list of things that need to happen to get this into the repository, I could get someone from our team to do the work; we've been creating a bunch of scalars, so we could schedule this in our usual sprints, and get someone from our team to do the work. It depends how much work there is, and we need a clear list of tasks that we can do.
- Lee: nice; this is why we need to figure out what the mechanism is for people to submit stuff. I'm concerned that lots of people will open PRs to add scalars and there'll be no-one there to review them. The Foundations there to provide a safe harbour for IP, but Gabriel's right that we are volunteers so it's up to us to make things happen.
- Lee: if we reach out I'd not be surprised if we found volunteers to make this happen.
- Ivan: We need Andi to be involved in the conversation. Can we schedule a call to talk to Andi and have a conversation about how to proceed?
- ACTION - schedule a call with Andi (hour and a half)
- Who's interested?
- Lee
- Indicate your interest in the #wg channel on the GraphQL Slack
- Issue comment
- Courtney: [sharing screen] Some time ago a ticket was created about adding namespaces to the GraphQL spec. Various suggestions were put forward on the syntax, and why it should be added in the first place. The one thing that's clear is that the issue is contentious; I'm not going to argue for a particular syntax but for the usefulness of namespaces in general
- I can't speak for other people, so I'm going to try and clarify our use case at Hypi.
- Hypi allows people to create virtual apps, creating a GraphQL and REST API for you. We have the ability (just like in npm, etc) for apps to have inter-dependencies. These aren't just from your team/org but also from other people on the platform. So you share their types into your app.
- Currently we're doing hacks to avoid name collisions; we're thinking that namespaces would make this a lot simpler for customers.
- The common argument is that prefixes and suffixes achieve the equivalent solution without requiring namespaces. This is a valid solution in many cases.
- In our app we have realms, with releases and (something). So we'd need to use all these values to generate the type name to ensure uniqueness. Names get long but it's okay.
- When it comes a problem is when you have dependencies between different versions, which cause conflicts. There's no way to resolve these type name conflicts; so you have to have really really long type names.
- This would be okay for Java clients to generate these automatically; but our customers are using GraphQL introspection and dynamically using the results.
- Because we're generating the type and function names, it becomes very unreadable. Generated names can't even be seen in GraphiQL in many cases because they're so long.
- The existing solutions aren't impossible, but they're very painful to work with.
- Secondly, other than name resolution, we provide authentication methods in the schema. E.g. we can protect Query.xyz with a permission scope. These explode; and the authorization policies become very brittle. The solutions aren't maintainable by hand, it needs to be automated, and our customers don't have the resources to handle this.
- Our customers don't have the tools to deal with this complexity; they can't use the generators, etc.
- We've submitted a description of how we solved this, as requested by Lee, and there were no major problems. The modifications to GraphQL Java only took 2 days. What became the real problem is when customers started using it, they could not use the GraphQL ecosystem with it because Apollo, etc doesn't support them. We've not rolled it out broadly; but it does make life a lot easier for our customers.
- I do think there's a lot of benefit, especially for our customers, in adding namespaces
- (in chat) Ivan: GraphQL Import is pretty popular in ecosystem https://www.npmjs.com/package/graphql-import and we can use similar approach
import Foo from “./somewhere” prefixWith “SomePrefix”If we add top level directive as we previously discussed it can done natively in SDL - Courtney: We modified GraphQL Java in the example implementation
- Lee: my concerns weren't about complexity in the SDL (this isn't generally a big concern); my concerns are what happens to clients that don't support namespaces. Namespaces are not a simple thing across other languages.
- Courtney: in my comment I've laid out solutions to this including an algorithm for default namespaces. If you don't specify a namespace then things go into the default namespace. So if a schema doesn't have namespaces they just fully drop into the default namespace. None of the customers had to change their existing schemas to support this feature - schemas are backwards compatible.
- Evan: namespaces would also be useful for us, Shopify's schema has got very large, prefixes are in use right now and it's not the most pressing issue right now, but it'd be a nice to have.
- Gabriel: Can you describe the reason for the issue?
- Evan: our schema is enormous internally (much larger than our already large public schema); keeping names distinct when they refer to similar things across the company can become complex (like Java FactoryFactory naming). It works, it's not a huge problem for us, but it's awkward and namespaces would make it easier.
- Andrew: we have user-defined schemas and we're using prefixing; namespaces would be good for us too, so +1 from us.
- Evan: if we were to convert our schema to namespaces we'd have at most 2 or 3 layers. There's definitely cases where there's more than one layer.
- Courtney: For us, it’s the same. We also have three layers: customer realm name, app name, name of release
- Lee: wouldn't it be a breaking change to change the type name based on the release?
- Courtney: yes, but it's going across these versions that's the challenge.
- Lee: Most of the original motivations for namespaces were not very convincing (fragmenting the schema within an organization for example). But this use case is way more compelling. A truly distributed schema, intermeshed interleaved schema across different companies and organizations. Tying GitHub with Shopify for example, namespaces would be the way to allow that.
- Courtney: we have functionality that lets customers do exactly that using an OpenAPI specification - we'll automatically build queries and mutations from those. Customers using our services for this causes an explosion of type names; even just the Stripe API is humongous, so combining multiple sources can make it unusable in many cases.
- Ivan: this is not a new problem; we've discussed it with Prisma guys, and they made graphql-import. They used comments - "import <typename> from <...>"; you can also add custom prefix - like in JS you can "import { Something as SomethingElse }"; in GraphQL you can't do this exactly but you can use a custom prefix. In the case where you want to import a few custom types we can allow people to use custom prefixes. We don't need to let them use generated prefixes, they can choose whatever prefixes they want. See my notes in chat.
- Courtney: in the comment on that issue, we added support for aliases and partial imports; we added support so you could import a list of types from a namespace; also allowing "import namespace.Type as Name" to allow aliasing.
- Ivan: instead of importing namespaces, allow people to provide prefixes.
- (in chat) Evan: concretely, we have this type in our schema right now: MerchantInsightsOnlineStoreConversionFunnelDataPoint
- Courtney: one thing is figuring out if the GraphQL community want to take this forward; we're keen for it, so we'd be happy to write the RFC. We need some action items to move this forward. A) is there any strong reason not to have this - if so, can detractors explain? B) If not, what's the next steps - is it an RFC?
- Lee: Personally, I'm concerned that it adds a layer of complexity, and there are reasonable workaround even though they have downsides. Next step would be an RFC with a clear explanation of what the problem statement is; make it clear that existing solutions are inadequate; how likely is it that the typical GraphQL user will use this. We don't want to accumulate a lot of different fringe features in GraphQL. Make it clear that this is desired by a large portion of GraphQL users. Then there's questions of backwards compatibility, degenerate cases, can you break a schema somehow; how can it bite us? What are the foot guns? Are we addressing all these things.
- Evan: the things you've raised today are a good first step here; it needs cleaning up and writing down in a way that will serve us for posterity.
- Ivan: we need to formalize the discussion. I don't have a GraphQL API personally; I'm an implementor. From an implementors point of view, it would be great to demonstrate how many changes there were for GraphQL Java - was it 100 lines? 1000 lines? There's a complexity budget, we want to make sure the changes are not too large for all the implementations to implement.
- Courtney: we have a fork of GraphQL Java internally; I'll have someone try and isolate the changes and share them publicly.
- Ivan: that would be great.
- PR
- Benedikt: I bring this up because there was relatively little feedback. I added this criteria about developer ergonomics with developing input unions. They place some restrictions on which input types can be combined and how.
- Lee: I merged your PR. Ultimately, we’re ruling out option 4 because of this constraint. If I can recap, one of the things we thought about was schema evolution with these new features. For 4 (structural uniqueness) in particular, you could end up in a bunch of bad situations. We felt that it boxed you into too many corners.
- Benedikt: That makes perfect sense. I came up with another example that couldn’t be used for 2, 3, 4. Combining input types in multiple input unions can be tricky or impossible.
- Lee: 4 in particular is very seriously affected by this. 3 it could potentially have an effect. 3 as it’s posed right now is not enough, so we’re currently trying to combine solution 1 and 3.
- Benedikt: One aspect of 3, the way the algorithm works always has a way to determine a match, sure, but it’s still fuzzy in a way. Most of the time, that’s exactly not what you want, to have your schema interpreted by the backend. Could lead to serious bugs.
- Lee: Yes, I agree. If it’s difficult to know how the schema should be interpreted, there is no guarantee that the error messages will be useful.
- RFC
- Lee: Benjie, do you want to give an update?
- Benjie: As Lee said already, we discussed all the other solutions. We ruled out 4. We are discussing 6 which is a combination of 1, 2, 3. Talked about nested complexity which can be introduced if you are not describing the type explicitly. Especially if you have a tree like type structure. Basically, the difference between nominative typing (explicitly this thing) vs structural typing. The structural typing is flexible but it is a lot harder for the backend to implement and could negatively affect performance.
- Solution 5, which is the @oneOf directive. This might not be the right direction to go either. It will never have arguments. Might not have support for directives. It has custom handling of nullability and we might need to expose that. If we can reuse the way that we currently handle nullability, that would be great. Rename these to “tagged output unions” and give it a concrete new type in the spec.
Sorry, it is storming in my area and my internet cut out so I missed a few minutes of conversation
In retrospect, the following notes seem to be related to the last topic of the day
- Lee: I would like to discuss the impact of tooling. Auto completion is such an important aspect to GraphiQL for example.
- Benjie: The tagged union would be very easy for beginners to use. One of the big things I have to answer on the RFC is when to use the tagged version vs the regular union type.
- Vince: I was thinking about symmetry between regular unions and tagged unions. What if we added a new type called a struc that could be used for both inputs and outputs.
- Benjie: Structs are an interesting idea but it prevents us from doing stuff like field relations (based on a User, find his friends) or if we support it, there would be a lot of added complexity.
- Benedikt: Have you thought about the migration path for a user who is using the regular union and would like to use the tagged union types?
- Benjie: Currently, no. If you already have tagged unions, then you can convert to the new directive based ones very easily. I’m at the moment thinking it’s going to need to be a new field and we will have to deprecate the old one.
- Benedikt: There already might be users who implement tagged unions, just without strong validation. Whereas on the other hand, adding a new discriminator field, nothing like that exists formally.
- Lee: The tagged unions is already a common practice. We should consider for people who want this feature, how are they already working towards that goal?
- Benedikt: If an input union is reduced to contain only one possible type, the schema is unnecessarily complex, and then you want to bring it back a level.
- Lee: Going from informal tagged unions to formal ones is the way to go. I think it is inevitable. Either way, we face some kind of breakage, but right now, it seems that the tagged version is the most promising in terms of ease, simplicity, and adoption.
- Benjie: Another problem we found with structural typing was the JSON scalar, where they actually use the JSON object as a scalar, which seems to be a common practice. This makes the structural typing somewhat incompatible.
- Lee: Yeah, you would have to do some complex stuff to deal with JSON scalars like trying to evaluate scalars last.
- Ivan: I'm concerned about using "tagged union" as the name since we already have unions. I was originally concerned that "one of" added another layer and I didn't like that; but it seems a lot of languages do this already so it's okay.
- I think boxing is a necessary price to pay
- Benjie: I’d be interested in discussing with that separately with you, Ivan
- Lee: I think we should be careful with the naming, as Ivan said. “Union” and “tagged union” can lead to confusion. I think the naming for this is up in the air and we should make that clear in the RFC. There are also enum types, and we just reinvented Swift enums and it would not make any sense to merge these together. Some languages call this exact type an enum.
- Benedikt: I would like to point out what Benjie said that was useful, I like the example of the name of type as the field names, in a way, what we really want to do is to uniquely identify the types in the union, and that’s exactly what type names are useful for.
- Benjie: I also liked that approach but I also liked not having the approach. The nice thing about the approach is the symmetry between input and output. We can potentially have two different tagged unions with the same name but for different usages. I think for now, we are going with the named version.
- Lee: I recommend the named version instead of the typename, the first one is for semantics. The name gives you a tool to help you describe what you’re looking at. A nice thing with the tagged union is you can include a type more than once. Hasura has a complex data filtering algorithm, 30 different things you can use, it is essentially a tagged union, but a number of the options are of the same type. In previous versions, it was like defining a type with a single field in it and then creating a union out of all these types.
- Lee: This was mostly meant to be an update. The next step is to draft an RFC that contains everything we discussed.
- ACTION - Draft a new RFC that contains everything we discussed
- Ivan: I have some ideas for @oneOf. To put those ideas in the same document would create more confusion. Where should I put these ideas?
- Benjie: Perhaps add it as a new solution in the RFC, for example solution #7. We can discuss in private if you want if you are unsure if it is unique enough.