Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Project Descriptor(project.toml) Converter #182

Closed
Closed
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8435e34
draft
haliliceylan Aug 10, 2021
a6ab15c
last
haliliceylan Aug 10, 2021
429c841
Update text/0000-project-toml-converter.md
haliliceylan Aug 12, 2021
4e93a27
CNB_PLATFORM_API
haliliceylan Aug 13, 2021
93b2bd0
motivation prepare phase
haliliceylan Aug 13, 2021
ca270f3
motivation paraphraph 2
haliliceylan Aug 13, 2021
f1aa294
What it is
haliliceylan Aug 13, 2021
9002b56
natalie author
haliliceylan Aug 13, 2021
b56261d
project toml example
haliliceylan Aug 15, 2021
bde82fa
some changes
haliliceylan Aug 15, 2021
86e16cb
Update text/0000-project-toml-converter.md
haliliceylan Aug 16, 2021
077ef6f
Update text/0000-project-toml-converter.md
haliliceylan Aug 16, 2021
ee7363c
Update text/0000-project-toml-converter.md
haliliceylan Aug 16, 2021
a1b591d
Update text/0000-project-toml-converter.md
haliliceylan Aug 16, 2021
54acf19
Apply suggestions from code review
haliliceylan Aug 16, 2021
bad9c48
update project toml
haliliceylan Aug 18, 2021
aa3611a
platform schema
haliliceylan Aug 18, 2021
047ef8b
Update text/0000-project-toml-converter.md
haliliceylan Aug 20, 2021
be67fa4
Update text/0000-project-toml-converter.md
haliliceylan Aug 20, 2021
58c1f28
Update text/0000-project-toml-converter.md
haliliceylan Aug 20, 2021
a053a83
last feedback
haliliceylan Aug 20, 2021
f5bddf3
little md fix
haliliceylan Aug 20, 2021
17487a4
add some values
haliliceylan Sep 14, 2021
06393bd
natalie suggestion
haliliceylan Sep 14, 2021
9b0c441
natalie suggestion
haliliceylan Sep 14, 2021
5c2075d
Update example
natalieparellano Oct 1, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions text/0000-project-toml-converter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# Meta
[meta]: #meta
- Name: Project TOML Converter
- Start Date: 10 Aug 2021
- Author(s): haliliceylan, natalieparellano
- RFC Pull Request: (leave blank)
- CNB Pull Request: (leave blank)
- CNB Issue: (leave blank)
- Supersedes: N/A

# Summary
[summary]: #summary

The idea is to ship a binary with the lifecycle that would be responsible for translating project.toml from the schema defined in the project descriptor extension spec into something that the lifecycle knows the platform can understand i.e., a schema defined in the platform spec.

# Definitions
[definitions]: #definitions

* __project descriptor__ - a file that is included in the application directory to configure various behavior related to buildpacks; for more information, see the [`project.toml`](https://github.com/buildpacks/spec/blob/main/extensions/project-descriptor.md) extension specification

* __CNB_PLATFORM_API__ The Project TOML Converter will use the `CNB_PLATFORM_API` environment variable to decide the target schema. The `CNB_PLATFORM_API` environment variable is set by the platform in the lifecycle's execution environment. Version 0.3 is assumed if it is not set.

# Motivation
[motivation]: #motivation

- This feature has similarities with the "prepare" phase that has been discussed previously (see #555). But it is much smaller in scope. Potentially, the binary described in this RFC could eventually take on more responsibilities and look more like "prepare". But that is out of scope for this RFC. If the binary were to eventually take on more responsibility, all of its functions should be easily turned on and off, so that platform operators could choose which functionality to expose to users.

- Project Descriptor (aka project.toml) is a file which describes behavior related to buildpacks. It is non-trivial for platform operators to upgrade support for different versions of project.toml. So with this feature we are giving to developers the ability to use any version of project.toml knowing that the platform can always support it. At the same time we are giving to operators the convenience of only having to know about one schema at a time for project.toml.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What operators are we talking about here? I have the sense that this should say platform implementers. If that is so then I have a predicament with this statement. Only a subset of the project.toml wouldn't be translated due to the fact that the spec allows for undefined keys/tables. It seems like platforms may very well still have to care about the initial input format to some degree.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


- What is the expected outcome?

# What it is
[what-it-is]: #what-it-is

The Project TOML Converter is a CLI tool that will ship alongside the lifecycle. It has only one responsibility at this time, to support conversion between schemas. The tool will not need to be executed in a containerized environment (similar to `rebase`). Though it can be invoked as a separate binary it may end up being part of the lifecycle which itself is a multi-call binary (see https://github.com/buildpacks/rfcs/blob/main/text/0024-lifecycle-multicall-binary-for-build.md).

# How it Works
[how-it-works]: #how-it-works

Example Project.TOML with 0.1 schema
```
[project]
id = "io.buildpacks.my-app" # machine readable
name = "my app" # human readable
version = "1.2.3"
authors = ["Great Developer"]
documentation-url = "my-site.com/docs"
source-url = "my-site.com/source"

[[project.licenses]]
type = "some-license-type"
uri = "my-site.com/license"


[build]
include = [
"cmd/",
"go.mod",
"go.sum",
"*.go"
]
exclude = ["secrets.yaml"]

[[build.buildpacks]]
id = "io.buildpacks/java"
version = "1.0"
uri = "myreg.io/myspace/java"

[[build.buildpacks]]
id = "example/post-build"

[build.buildpacks.script] # does this make sense?
api = "0.5"
inline = "./post-build.sh"

[[build.env]]
name = "SOME_VAR"
value = "some-val"


[metadata]
cdn = "https://cdn.example.com"
```


Example Project.TOML with 0.2 schema
```
[_]
schema-version = "0.2"
id = "io.buildpacks.my-app"
name = "my app"
version = "1.2.3"
authors = ["Great Developer"]
documentation-url = "my-site.com/docs"
source-url = "my-site.com/source"

[[_.licenses]]
type = "some-license-type"
uri = "my-site.com/license"

[_.metadata]
cdn = "https://cdn.example.com"


[io.buildpacks]
builder = "myreg.io/myspace/builder"
include = [
"cmd/",
"go.mod",
"go.sum",
"*.go"
]
exclude = ["secrets.yaml"]

[[io.buildpacks.group]]
id = "io.buildpacks/java"
version = "1.0"
uri = "myreg.io/myspace/java"

[[io.buildpacks.group]]
id = "example/post-build"

[io.buildpacks.group.script]
api = "0.5"
inline = "./post-build.sh"

[[io.buildpacks.build.env]]
name = "SOME_VAR"
value = "some-val"


[com.fun-company]
name = "I am missing in the 0.1 schema version"
```

# Drawbacks
[drawbacks]: #drawbacks

Why should we *not* do this?

- Another binary to maintain

# Alternatives
[alternatives]: #alternatives

- Having the lifecycle process project.toml directly (like prepare phase see: #555). This RFC is only focusing on the upgrade problem so that it can be done quickly.
- Do nothing, hope everything will be fine (we can leave as it is). We think it's too hard for platform operators to support multiple versions of project.toml.
- Having the translator binary translate from e.g., project.toml schema 0.1 -> project.toml schema 0.2. The platform API already represents the contract between the lifecycle and the platform, so it makes more sense than "project descriptor schema -> project descriptor schema" translation..

# Prior Art
[prior-art]: #prior-art

Discuss prior art, both the good and bad.

# Unresolved Questions
[unresolved-questions]: #unresolved-questions

- What will the platform schema for project.toml look like?
- Out of scope: parts of project.toml that aren't relevant to the platform api (like setting env vars)

# Spec. Changes (OPTIONAL)
The new platform schema could look something like:
```
[project]
id = "<string>"
version = "<string>"

[build]
builder = "cnbs/sample-builder:bionic"
include = ["<string>"]
exclude = ["<string>"]

[[build.buildpacks]]
id = "io.buildpacks/java"
version = "1.0"

[metadata]
# all metadatas here
```

*Here are the namespace conversions*

For conversion to/from v0.1
```
project = project
build = build
build.buildpacks = build.buildpacks
metadata = metadata
```

For conversion to/from v0.2:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should call out explicitly that we only intend to "translate" data under _ and io.buildpacks. Then there is the question of what to do with data under other domains - I believe that

  • the original project.toml in 0.2 schema
  • a separate file for "other" stuff
  • a separate file for each domain - but this would be hard because we wouldn't know which . to split on

were all discussed. I don't think there is much substantive difference between the first two, and my preference is for the first one for its simplicity.


```
project = _
build = io.buildpacks
build.buildpacks = io.buildpacks.group
metadata = _.metadata
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to overlapped properties? ie. translating the following from v0.2 to v0.1:

[_]
name = "My App"

# in this schema version this is consider arbitrary/user data
[project]
name = "Client 2 Project"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can add a global prefix or postfix for keys which not included this list ? (just an idea)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5c2075d should help clarify the mapping from 0.1 <-> 0.2 ...though I wasn't sure what to do about [build.buildpacks.script].

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, we were going to ignore (not map) previously non-existent properties. It doesn't make sense for platforms to expect properties that aren't associated with their specification.

```