Code generation with cue

[cueckoo]

Originally opened by @myitcv in cuelang/cue#1027

Originally posted in Slack by @eadlam:

I’ve been exploring code generation with cue for the last 6 months. I’ve built a configuration system which defines types in cue and exports those types to capnp. Then I refine those types to fully specified configs which I materialize to json. The generated capnp cpp classes can then load the json.My code generation pipeline is cue->openapi->capnp->cpp. I generate the capnp specs from openapi using Python.In general, I think the ability to represent the cue AST in json is extremely valuable, as it provides a language agnostic interface for code generation.Since OpenAPI is not fully capable of representing the cue AST, I think it would be great to move forward with generating a cue-specific json schema.My next project is to model a whole system in cue and generate state machines and potentially TLA+ code from that model. I believe cue can model systems to the same degree that SysML can, and may provide a viable code-first approach to model based software engineering. A fully compatible json schema would be a huge enabler for this effort.

---

Response from @mpvl:

Open to proposals how that would look like.

---

Response from @myitcv:

@eadlam is the AST by itself a sufficient interface for code generation? My suspicion is that it would not be.

---

Response from @eadlam:

With the current OpenAPI export, I have access to types and partially specified dependent types. With that I believe I can generate hierarchical state machine scaffolding, including states, event types, simple guard functions, pre/post conditions and at least the method signature for more complex action/guard functions.

Although that still leaves the action/guard functions for developers to implement, it’s still a sizable chunk of executable code which also happens to be at the same level of abstraction one needs for model checking.

The remaining actions/guards that the developer implements would be simple functions that take a mutable state object and immutable event object, and update the state in some way. Cue’s partially specified types can then be used to generate tests that the state properties remain within correct bounds.

As I said, I think the OpenAPI spec is actually sufficient to do all this. I imagine I could do more with the AST, but I don’t know that for sure.

I guess there are really two separate things that I'm interested in:

1. Schema definitions exported to regular json schema if this would allow us to provide more detail than what we currently get with OpenAPI. I thought I remembered @marcel van Lohuizen saying that cue can't be completely mapped to OpenAPI, but I don't know if json schema itself provides a way around that
2. The type/value lattice in some json format

[magick93]

a viable code-first approach to model based software engineering.

This is the value I see in cue.

Previously I've used Eclipse Xtext and Xtend to develop DSLs, from which I could then generate code for various layers (c++, protobuffer definitions, UI components etc).

Combining Cue with something equivilant to Eclipse Xtend would allow for generating basically any kind of file - json, sql, etc

[verdverm]

That's what I'm working on with https://github.com/hofstadter-io/hof

CUE + text/template

The hard part is coming (or more so standardizing) up with a schema + generators, such that it allows the users to be technology agnostic (it works for many). hof is a general engine for developing CUE based schemas and the templates they are combined with.

[magick93]

Thanks @verdverm - hof looks excellent! Will try it today!

[orpharion]

I've created a rough command line tool for dumping AST to JSON, and compiling CUE source from JSON, here:
https://github.com/orpharion/cue-extensions
The CUE AST nodes are all unique, so you can determine what a node is entirely by its form - you don't need to keep kind or type information.
I'm using it to transcode the structure of dotnet libraries to CUE:
https://github.com/orpharion/CueFSharp
I find the approach works really well without involving JSON schema

[eadlam]

Update

I'm having a lot of success using Cuelang as a modeling language to define state machines (still using the cue->json-schema approach). From these models I'm generating executable cpp, diagrams, and TLA+ code for model checking and formal verification of temporal properties.

There are two features that would be nice improvements to my cue models:

1. boolean not (!) applied to structs
2. allow an (extraneous) leading conjunction/disjunction symbol

1. boolean not applied to structs

Consider this snippet which shows how I'm composing guard functions:

#A: {a: {}, ...}
#B: {b: {}, ...}
#notB: {"!b": {}, ...}

#guard: #A & #notB

It would be a lot nicer if I could do:

#A: {a: {}, ...}
#B: {b: {}, ...}

#guard: #A & !#B

In cue, this would mean that #guard is not allowed to include b: {}. In json-schema, this would use the not keyword for that property. I've noticed cue sometimes generates json-schema with not, but I think it's only when a disjunction includes an explicitly closed struct.

2. allow a leading conjunction/disjunction symbol

This would allow long conjunction/disjunction expressions to be formatted like bullet lists. TLA+ has this and although it's a seemingly minor syntax sugar, it is really nice in practice.

For example, instead of this:

#guard: {#A & #notB} |
        {#A & #B & #C} |
        {#C & #D}

Being able to do this:

#guard: | {#A & #notB}
        | {#A & #B & #C}
        | {#C & #D}

And even better if we could do this:

#guard: | & #A
          & #notB
        | & #A
          & #B
          & #C
        | & #C
          & #D

cc: @mpvl @myitcv

[mpvl]

@eadlam: first let's address 2:

I see the appeal of such a formatting. It will be very tricky though, for various reasons.
For consistency, it would then make sense to also allow such a notation for other operators. But this results in an ambiguity:

#foo: +3
+5

Using the proposed, this could be expected to mean #foo: 8, but currently it will be an embedded scalar { 5, #foo: 3 }.

Furthermore, the proposed syntax would ideally be an expression (especially considering the indicated desire to have this nested), but this doesn't play well with the comma elision mechanism. Consider for instance:

a: | 1
   | 2

   | 3
   | 4

Would this be equivalent to {a: 1|2|3|4} or {a: 1|2, 3|4}? There is clearly an ambiguity.

The only thing I could imagine is to lead with some kind of escape character, like

a: \
   | 1
   | 2
   | 3
   | 4

vs

a: \
   | 1
   | 2
\
| 3
| 4

This seems quite awkward to me, though.

Maybe a compromise solution is to allow \ to be used to bypass comma elision:

a:   1 \
   | 2 \
   | 3 \
   | 4

vs

a:   1 \
   | 2

3 \
| 4

This is more verbose, but the mechanism would be more widely applicable. It is a bit ugly, but at least it has fairly unambiguous semantics.

This latter use would also be consistent with the use of \ in multiline strings (analogous to Swift strings, but currently not supported).

[eadlam]

I'm in favor of having not(), and I think ! would be the appropriate operator to pair with it. That would be consistent with |==or() and &==and().

[verdverm]

Note that & and | are binary operators, whereas ! is unary.

a: bool
b: !a

does this mean b is not a bool or the negation of a?

[eadlam]

Your last example is supported by cue, and it means negation of a. It throws an error if b is not a bool.

I think in a way cue already has some logic for processing what not means at a struct level. For example, consider this:

#guard: #A: {}
#guard: #B: {}

let A = {a: #guard.#A}
let B = {b: #guard.#B}

#transition: (A & B) | A

Exporting this to json schema produces this:

...
...
{
    "guard": {
        "type": "object"
    },
    "guard.A": {
        "type": "object"
    },
    "guard.B": {
        "type": "object"
    },
    "transition": {
        "type": "object",
        "oneOf": [
            {
                "required": [
                    "a",
                    "b"
                ],
                "properties": {
                    "a": {
                        "$ref": "#/components/schemas/guard.A"
                    },
                    "b": {
                        "$ref": "#/components/schemas/guard.B"
                    }
                }
            },
            {
                "allOf": [
                    {
                        "required": [
                            "a"
                        ],
                        "properties": {
                            "a": {
                                "$ref": "#/components/schemas/guard.A"
                            }
                        }
                    },
                    {
                        "not": {
                            "anyOf": [
                                {
                                    "required": [
                                        "a",
                                        "b"
                                    ],
                                    "properties": {
                                        "a": {
                                            "$ref": "#/components/schemas/guard.A"
                                        },
                                        "b": {
                                            "$ref": "#/components/schemas/guard.B"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        ]
    }
}

This implies that cue's interpretation of A & !B would be A & !(A & B)

[verdverm]

Your last example is supported by cue, and it means negation of a. It throws an error if b is not a bool.

Yes, this is the current semantics. I was referring to your proposal that ! as shorthand for not(...).

I'm in favor of having not(), and I think ! would be the appropriate operator to pair with it.

So in the case of a boolean, the ! would be ambiguous in a RHS context, we could not infer if it means negation of a boolean or not(boolean).

[myitcv]

My only thoughts based on the discussion above are:

• Are we perhaps jumping to thinking about syntax and language changes too early?
• Are we clear that the problems we're discussing here are definitely the right problems to be solving?
• Does the proposed not() builtin achieve what's required here?