Reading external pkl resources as pkl objects

[mikea]

[Forgive me if this is something basic, but I can't find an answer in documentation]

I am trying to write pkl document transform. It want it to be used like pkl eval -p input=my_input.pkl transform.pkl where I always expect my_input.pkl to amend a certain MyRootModule module.

How do I read() a typed value from the input then? I have used read(input) to receive a Resource object with untyped text, but how do I get the instance of MyRootModule instead?

[bioball]

read() expressions can only give you values of type String or Resource (which ultimately still gives you a string).
If you want to then turn that string into a typed object (like in the case of reading JSON), you'd then need to parse it then coerce it into that type (e.g. with deepToTyped).

In your case, though, since your resource already is a Pkl module, you should instead just import it.
Inferring further: if you want to selectively import a module based on an external property, you can do this with the help of a glob import.

import* "**.pkl" as allModules

theModuleIWant = allModules[read("prop:input")]

A glob import gives you a mapping whose keys are the expanded glob pattern, and values are the matched module itself (see the language reference) to learn more). This gives you the effect you're looking for, where pkl eval -p input=my_input.pkl transform.pkl can give you a MyRootModule inside transform.pkl.

Note: relative glob patterns are expanded relative to the module that the glob is declared in, not your CWD.

Note 2: thanks to Pkl's laziness, import* "**.pkl" doesn't actually import all the matched modules unless they are needed. In this example, it only executes an import of the selected module, because only that one is used in the program's output.

[mikea]

@bioball thanks for the suggestion. I am not sure how would glob approach work with input files not in the source tree?

I'd like the following to work:

• pkl eval -p input=/some/generated/file transform.pkl
• pkl eval -p input=local.pkl /pah/to/transform.pkl

As well as supporting something along the line: pkl eval -p input=local.pkl http://my.domain.com/pkl-transform@1.

Your suggestion gives me an idea to generate temporary root .pkl file that will import everything I need, but I wonder if there exist a pkl-only solution?

Maybe import(read("prop.input").uri) would work?

[bioball]

Pkl's imports must be static strings, so, import(read("prop.input")) wouldn't work.

What's the use-case for wanting a dynamic import like this?

One other solution could be to use modulepath. For example, in transform.pkl:

import "modulepath:/foo.pkl"

Then, you'd provide a modulepath that contains foo.pkl:

pkl eval --module-path /my/module/path transform.pkl

The limiation here is that your module's name must be known ahead of time.

[mikea]

What's the use-case for wanting a dynamic import like this?

I am building a declarative model and several generators ("compilers") for it.

A close analogy would be:

• the user authors declarative list of services to deploy with some of their high level config
• different generators generate all the necessary files for deployments depending on different needs: say kube configs or systemd files, aux config files, documentation, diagrams etc.

I have written prototype generators in go and realized that pkl itself suits extremely well to template-based generation.

The way it works currently (with stand-alone generator):

• user amends a module that has typed resources properties
• the tool loads pkl using pkl-go
• the tool generates all necessary files using go templates.

I really want to get rid of the go tool and to being able to "evaluate" external pkl files using pkl itself. I was able to convert all templates and it is a breeze and very pleasant. -p input=/path/to/my.pkl or similar to get typed input is what I can't figure out.

The only workaround that I know is to bring in all possible generators into service definition itself, but that is not ideal since I want to have them independent and potentially there could be many generators (including 3rd party).

[HT154]

I've run across this class of problem before and solved it by inverting this problem in on itself. At a high level, the idea is to have the user module be the entrypoint to evaluation (which you're already doing) but do transformation/templating selection a different way.

If I were building your package, I might do something like this:

Template.pkl

abstract module my_thing.Template
resources: <something>

kubernetes.pkl

extends "Template.pkl"

// do all the transformation and rendering needed here

systemd.pkl

extends "Template.pkl"

// do all the transformation and rendering needed here

Entrypoint.pkl

module my_thing.Entrypoint

import "Template.pkl"

resources: <something>
templates: Mapping<String, Template> {
  ["kubernetes"] = import("kubernetes.pkl")
  ["systemd"] = import("systemd.pkl")
}

output = (templates[read("prop:template")]) { resources = module.resources }.output

Then consumers of your package would amend Entrypoint.pkl as myEntrypoint.pkl

amends "@foo/Entrypoint.pkl"

resources { ... }

And evaluate via pkl eval myEntrypoint.pkl -p template=kubernetes

Taking this further, a user could supply their own templates, like so:

myTemplate.pkl

amends "@foo/Template.pkl"

// do all the transformation and rendering needed here

And adding to the above myEntrypoint.pkl:

templates {
  ["myTemplate"] = import("myTemplate.pkl")
  ["thirdParty"] = import("@thirdParty/template.pkl")
}

To use this template, the user would run pkl eval myEntrypoint.pkl -p template=myTemplate

[mikea]

Yes I understand this inversion, but it is undesirable:

• it creates strong coupling
• prohibits generators that are not included. What if someone needs a terraform generator?
• adds generators that don't make sense. E.g. following the service analogy it either makes sense to generate systemd files if you are targeting local deployment. Or it makes sense to generate kube files if you deploy to kebernetes. It doesn't make sense to include all possible outputs when you don't need them.

Hope it makes sense?

[mikea]

BTW I have tried generating temporary

import "$input" as input
import "$generate_home/generate.pkl" as generate

output {
    text = generate.generate(input.resources)
}

but this creates another problem when I try to cast values in my generate.pkl:

Module version conflict: Expected value of type `mymodule#Package` defined in module `projectpackage://mydomain/[email protected]#/main.pkl`, but got type `mymodule#Package` defined in module `package://same_url`. 

Because my generate.pkl tool uses PklProject to reference mypkg this creates problems? Should I remove PklProject from the tool?

[bioball]

This happens when you mix project dependencies with absolute package imports. You'll need to stick with one or the other.