RFC: Project Descriptor(project.toml) Converter

[haliliceylan]

Readable

Hello everyone,

This is my first RFC as a GSOC Mentee at this summer. We prepared this RFC with @natalieparellano. We spent some time in my own fork to get ready this RFC. You can access previous comment and feedbacks from here.

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[haliliceylan]

for DCO I force pushed. and fixed conflict errors because of force push.

[jromero]

for DCO I force pushed. and fixed conflict errors because of force push.

Looks like you might have accidently signed too many commits back. The commits tab should only include your changes.

[haliliceylan]

for DCO I force pushed. and fixed conflict errors because of force push.

Looks like you might have accidently signed too many commits back. The commits tab should only include your changes.

Thanks for making me realize, To fix problem, I runned these commands one by one;

# remote/main -> this repo, remote/origin -> my fork repo
git checkout -b backup_3 origin/project-toml-converter # backup for better way
git rebase main/main # i discard any conflict except rfc file
git reset main/main # Keep files but delete all previous commits
git add . && git commit --signoff -m "blabla" # create one commit with signed off

I hope problem is fixed. Please let me know if there is any better way to protect branch history.

---

Edit: I found better way;

git reset --hard b27bcb2b025c661e8694a51db70f342d1d710324
git rebase main/main --signoff
git push --force-with-lease origin project-toml-converter

[aemengo]

The Project TOML Converter is a CLI tool that will ship alongside the lifecycle.

Why are we deciding to make this a separate binary? As opposed to another code path that gets run? Is this a preemptive move for a ./prepare phase binary?

--

I guess... I'm not sure that this is a problem that has to be solved. While I agree with the underlying premise:

We think it's too hard for platform operators to support multiple versions of project.toml.

the behavior that I'm expecting is that the platform complains/rejects about deprecated project schema versions. My thoughts:

• There's plenty of other tooling that exhibit this behavior.
• Having the internal component of the platform silently upgrade my input not only feels like black magic, but a bit over-engineered.
• A future schema version might indeed be a breaking change, which would require user input before making a schema upgrade
• And finally, keeping in mind that the application developer is writing these things, I'm of the opinion that they should be prompted to upgrade by the platform. Otherwise, this RFC encourages this non-best-practice of using schema v0.1 forever.

But this is a good discussion that we should be having. Thanks for raising this RFC, @haliliceylan

[jromero]

RE: @aemengo's comment

The Project TOML Converter is a CLI tool that will ship alongside the lifecycle.

Why are we deciding to make this a separate binary? As opposed to another code path that gets run? Is this a preemptive move for a ./prepare phase binary?

This is a valid question, the only input I have from a platform's perspective is that we'd want to get access to only this conversion operation without any other side effects.

As an example, take the field builder on schema version A vs B, as a platform I'd like to call this operation (more about how to access it later) and get a schema version that I understand. I wouldn't want to parse and support multiple schema versions myself. Also, I wouldn't want the conversion to try to do anything else that preparer may try to do.

How to access the functionality. There are a couple options for this:

1. For a platform like pack, we can use the lifecycle as a library and execute this conversion before running the build.
2. For a platform like tekton, we can use a lifecycle image as part of a step to call a conversion binary (or entrypoint)

I guess... I'm not sure that this is a problem that has to be solved. While I agree with the underlying premise:

We think it's too hard for platform operators to support multiple versions of project.toml.

the behavior that I'm expecting is that the platform complains/rejects about deprecated project schema versions. My thoughts:

• There's plenty of other tooling that exhibit this behavior.

Not sure I find that as a good justification.

• Having the internal component of the platform silently upgrade my input not only feels like black magic, but a bit over-engineered.

Based on my desired mentioned above (see "How to access the functionality") I'm not sure I'm seeing a lot of black magic.

• A future schema version might indeed be a breaking change, which would require user input before making a schema upgrade

Could you provide an example? Anything I can think or we've experienced in the past has had reasonable defaults or ways to mitigate.

• And finally, keeping in mind that the application developer is writing these things, I'm of the opinion that they should be prompted to upgrade by the platform. Otherwise, this RFC encourages this non-best-practice of using schema v0.1 forever.

The problem that I think we're trying to solve is the need for platforms to all be "in sync". This sort of functionality is to provide more interoperability between platforms. For example, I want my app to work (using project.toml) on pack, tekton, GCP, etc. It would be a bad experience if GCP told me to use schema A but pack said schema B.

[aemengo]

RE @jromero's comment

These are great points in response to my original comment. Thank you for raising them.

In general, once more, perhaps I don't see the "the need for platforms to all be in sync". To me, a parallel to the project.toml is the AWS Cloudformation Template. The template itself has a Format Version, with only one valid entry. We can assume that AWS will not convert my template for me. This is the behavior I've come to expect - even if other platforms accepted AWS Cloudformation templates.

I just wanted to point my predilection based on conditioning as a cloud tool user. It looks like this project, and maintainers, has a better perspective on how platforms should behave. And that's completely fine.

[jromero]

What operators are we talking about here? I have the sense that this should say platform implementers. If that is so then I have a predicament with this statement. Only a subset of the project.toml wouldn't be translated due to the fact that the spec allows for undefined keys/tables. It seems like platforms may very well still have to care about the initial input format to some degree.

[natalieparellano]

@jromero is this what you had in mind? https://github.com/buildpacks/rfcs/pull/182/files#diff-d6828a110979d409ba620d1b80593932c1cea332756b3cc7f7c14922dfc0b16aR132-R133

[jromero]

what happens to overlapped properties? ie. translating the following from v0.2 to v0.1:

[_]
name = "My App"

# in this schema version this is consider arbitrary/user data
[project]
name = "Client 2 Project"

[haliliceylan]

maybe we can add a global prefix or postfix for keys which not included this list ? (just an idea)

[natalieparellano]

5c2075d should help clarify the mapping from 0.1 <-> 0.2 ...though I wasn't sure what to do about [build.buildpacks.script].

[jromero]

IIRC, we were going to ignore (not map) previously non-existent properties. It doesn't make sense for platforms to expect properties that aren't associated with their specification.

[jromero]

If I recall correctly, arbitrary data would be retained. Could we get an example of a 1-to-1 real complex example to get a better understanding on how the translation would work?

[haliliceylan]

@jkutner @hone, do you have any good examples for that ?

[natalieparellano]

5c2075d may help

[natalieparellano]

We should call out explicitly that we only intend to "translate" data under _ and io.buildpacks. Then there is the question of what to do with data under other domains - I believe that

• the original project.toml in 0.2 schema
• a separate file for "other" stuff
• a separate file for each domain - but this would be hard because we wouldn't know which . to split on

were all discussed. I don't think there is much substantive difference between the first two, and my preference is for the first one for its simplicity.

[hone]

@natalieparellano do you mind if I move this blocked dependent on the resolution of #191?

[natalieparellano]

Let's draft this as we can safely say it is superseded by #202