Consider Rust, Cargo and crates

[jkrech]

CMSIS-Pack and CMSIS-Build focus on assembler, C and C++ as programming language at the moment.
@0Grit is requesting in the context of #43 to consider support for target software written in
Rust: https://www.rust-lang.org/
As well as looking at the concept of crates and packages managed using Cargo.
Crate: binary or library shipped as package
Cargo: the Rust package manager: https://doc.rust-lang.org/cargo/
Going forward we can expect to see Rust being combined with C:
Information regarding interoperability between Rust and C can be found here:
https://docs.rust-embedded.org/book/interoperability/rust-with-c.html

[jkrech]

From: @0Grit
Do not forget about Rust. It is quickly gaining momentum. New engineers feel no sense of attachment to C and will drop it like a rock.

Please 🥺🙏 consider looking at how Cargo and crates can be either used for inspiration or used directly to manage packs. Extend it if you have to.

[jkrech]

From: @iomint
@0Grit : I do not see an issue in what is proposed here to handle a project from packs containing sw components developped in C language or in Rust. Do you ?

[jkrech]

from: @0Grit
cmsis-build abstract the underlying build tooling from cmsis-packs? If so I trust your assertion.

I'm strapped for time and am unable to do much more than skimming of the materials. Just wanted to make sure these things are at least mentioned in the conversation. I've seen people like @ilg-ul who have a wealth of knowledge and work put into projects like this, get ignored in the past.

from :@fred-r
Well, I would think that:

the proposal is based on CMSIS-Pack components, so we can have a component providing RUST code
when it comes to the build chain, the spec lets it open I guess. Today it is proposing GCC, IAR and AC6 as a compiler but I assume we may support a toolchain with both C and RUST support ?
or maybe more realistic would be to mix C project and RUST projects compiled with their own toolchain each ?

I assume the major point you highlight is how to use both cargo and CMSIS packs ?

from: @ilg-ul
After embracing the CMSIS Packs idea and trying to use them for my projects (long time ago), I finally realised that their design has different goals, that do not match my criteria, especially in providing layer separation and reduced complexity.

Thus I designed xPacks, which are a more generic version of the highly successful, industry standard, npm packs.

In terms of the first criteria, layer separation, source xPacks are completely neutral to build processes, languages, platforms, etc.

In practical terms, source xPacks are collections of files with static dependencies to other xPacks, the same as npm packs are nodejs modules with dependencies of other packages.

Satisfying dependencies is a separate step that is executed before the build, to bring in all dependent packages, sources or binaries.

As for the second criteria, reduced complexity, CMSIS Packs favour very large packages (I saw some weighting hundreds of MB), with many, many source files, grouped in lots of components, each with its own version.

On the opposite end, xPacks (and npm packs), favour many small packages, each package having its own version, which applies to all files in the package.

Another major difference is that the main CMSIS Packs repository is centralised and not automated (or at least it was not when I used it, it required someone to accept the packages, and to update the index, which is, most of the time, outdated with vendor packages).

xPacks, being in essence npm packs, can be stored on any npm compatible server, like npmjs.com, GitHub, etc.

My 2 cents...

from @fred-r
==> to me this is the way CMSIS packs have been used so far.
The spec does not force you to do so.
I agree that it would be a much better practice to design smaller packs not mixing very different things.
The pack is only the way to convey the CMSIS components : this is the important point, designing consistent components.

from: @0Grit
I highly recommend looking at @ilg-ul's x-pack project. https://xpack.github.io/
I've used it and it's a great experience. He has packs to install toolchains, cmsis, etc. A lot of it sitting on top of NPM's infrastructure too. (Cross development host platform to boot)

The only thing preventing it from widespread usage is that he is basically the only person working on the project.
You would think he had a team considering how extensive the project is.

It would bring a tear to my eye to see some of his work be used by open-cmsis-packs somehow. Even if only by inspiration; though just using xpacks by itself and adapting the open-cmsis-packs spec to it would save everybody some time.
Not sure about how or if xpacks handles the "zones" concept though.

from: @ilg-ul
If allowed, multi-component packages may present additional challenges. Imagine you have multiple versions of a package which defines multiple components, each with its own version, all different in each package.

For whatever reason, a project may decide that it needs one component from one package version and another component from another package version. I did not check the recent specs, so I apologise if this is nonsense, but can the proposed package manager handle cases like this?

I don't know what the zones are, but I did a lot of thinking and experimenting with packages, and my current understanding of the subject is that any design that requires the packages to be identified by anything more than a (scoped) name and a semver, is a design too complicated.

As for what a package is, according to the npm documentation, "a package is any of the following":

A folder containing a package.json file.
A gzipped tarball containing (1).
A URL that resolves to (2).
A <name>@<version> that is published on the registry with (3).
A <name>@<tag> that points to (4).
A <name> that has a latest tag satisfying (5).
A git url that, when cloned, results in (1).

I personally could not find a more generic definition.

[jkrech]

from: @fred-r
The example provided above (#43 (comment)) and referenced by @0Grit is clearly not a single package.
It is an application project made of several software projects.
Each software project is the composition of several CMSIS components (via several CMSIS packs).
I agree there is complexity but it only reflects the complexity of the systems we are dealing with.

I think we should not mix all topics:

I understand that you challenge the CMSIS-pack technology => to me x-pack covers this based on what you explained, even if I think that a "proper" usage of CMSIS-Pack should do the job...to be discussed by @jkrech and @ReinhardKeil
There is the project management aspect => here nobody said an application project should be a single pack

So I would recommend focusing this discussion thread on subject 2 and maybe @jkrech can redirect the 1st topic to the appropriate issue ?

from: @iomint
We are indeed deviating from the subject of this thread but these are interesting discussions for which we can open new issues thread or redirect to existing relevant ones.
During early discussion to set-up this open-CMSIS initiative, we have agreed to start from CMSIS-Pack (It has a certain footprint in the embedded with many MCU vendors publishing CMSIS packs) and see how to improve the spec and push an open reference implementation; @ilg-ul I invite you to look at the others issues - in particular #10 and #11 - to get an overview of the improvements we are looking for. It is great if there are good things pointing in the same directions from your xPack work that we can pick-up. It can helpful if you could have a look to #10 and #11 and flag to us the relevant elements of xPack.
About too large/complex (CMSIS) pack I fully agree with @ilg-ul concern and the feedback from @fred-r. How to favor (enforce ?) design practices to end up with "small" packs is a very good topic and will deserve a dedicated issue thread. We can provide examples/patterns and/or do something with a Pack checker but where to set the limit is a tricky question.

from: @ilg-ul
I took a quick look, but as long as you do not have a clearly layered design, and throw everything (dependencies, build, components, configuration, etc) into the same bucket, it is hard to flag any relevant differences.

[ilg-ul]

I don't mind having my comments copied here, but I fail to see how they relate to Rust, Cargo and crates.

Two more comments:

• GitHub has Discussions and Issues; I don't think that these are issues
• Issues with lots of different topics under a generic subject are not very productive

[jkrech]

Thanks for the feedback. Moved from issue to discussion now.
If you want to open a separate discussion about xPacks as another alternative packaging concept besides Cargo, feel free to do so.

[0Grit]

@jkrech What about @iomint's feedback regarding needing a new discussion about the 'size' of cmsis packs?

[0Grit]

I don't mind having my comments copied here, but I fail to see how they relate to Rust, Cargo and crates.

Two more comments:

• GitHub has Discussions and Issues; I don't think that these are issues
• Issues with lots of different topics under a generic subject are not very productive

Agree on all points.

See #40 to understand why I resorted to commenting in an issue.

[0Grit]

Regarding the topic:

https://docs.rust-embedded.org/book/interoperability/rust-with-c.html

[ilg-ul]

open a separate discussion about xPacks

if someone thinks I can be of any help, sure, ask specific questions and I'll do my best to answer, but otherwise I don't plan to interfere with your project.

[fred-r]

@ilg-ul : yes, could you please clarify this sentence statement

"I took a quick look, but as long as you do not have a clearly layered design, and throw everything (dependencies, build, components, configuration, etc) into the same bucket, it is hard to flag any relevant differences."

What do you call the same bucket ?
The fact that all packs go in the same CMSIS_PACK_ROOT repository ?
Otherwise the CMSIS Pack spec tries to organize the components with categories for files, conditions and versions.

For the layering, I think there is a proposal:

• CMSIS packs brings the transport layer to convey the SW components on your disk
• cprj allows grouping the components to define a SW project
  - clayer is an incomplete cprj allowing to pre-group components in a re-usable layer

This cprj is more a format for build (generated), the real format for describing the proejct is more the cproject as far as I understand.

[0Grit]

@fred-r can you create a new discussion specifically for discussing CMSIS-packs layering so as not to muddy the rus/crates discussion?
Glad to see all the engagement.

[fred-r]

Ok: see #50

[ilg-ul]

Well, if the design is already layered, then you should be fine.