Are Open-CMSIS Packs intended to be project dependencies, or content is still expected to be processed by a tool and the result copied into the project?

[ilg-ul]

I took some time to read several of the existing proposals, but could not find a clear explanation regarding how Open-CMSIS Packs are intended to be consumed.

In the initial Arm specs, the CMSIS Packs were actually an extension of MDK, expected to be consumed only with MDK, which selected which source files were to be included into projects. Later a similar functionality was added to Eclipse.

CMSIS Packs were not intended to be consumed as direct dependencies of projects, there were no standard ways to define such dependencies, no reliable download locations, etc. To make things worse, CMSIS Packs favoured huge packages, which made direct dependencies even more unrealistic.

Which way do you plan to go with this major design issue in Open-CMSIS?

---

FYI, In the npm world, and, by extension, in the xpm/xPack world, packages are by design project dependencies, and their content can be used directly during builds.

The dependencies are listed in package.json and a separate install step downloads all of them. The content is basically read-only and can be directly used during any subsequent builds.

Packages can be downloaded from various locations, like public/private npm servers, public/private Git servers, etc.

The common expectation for npm packages is that, once a version is released and published on the npm public server, to be immutable, so that references to it will remain valid on long term.

[fred-r]

Open-CMSIS packs are a way to deliver components.
These components can be used in projects via the csolution/cproject files and a project manager.

At the moment we are discussing 2 ways of using these software elements:

• one by referencing from the project source code located in packs stored in CMSIS_PACK_ROOT
• one by copying locally in the project the packs in use

In the project files, you can keep the "loose" identification of packs and components or provide a detailed versioning requirement.

The pack size is indeed a major point: we need "reasonable" packs.
This is something that can probably be discussed here:
#108

[ilg-ul]

cproject files and a project manager.

Since I did not see the definition of the cproject files, the question may be stupid, but lets assume we have a Git repo with a project which has a valid cproject file, referring to other Open-CMSIS Packs.

How would a GitHub Action look like in order to:

• download the dependencies,
• build and run several unit tests
• do this on all available platforms, or at least on GitHub hosted runners (macOS, Linux and Windows)

I'm mainly interested in the command to satisfy the project dependencies, the build itself can be anything (like cmake, meson, or even plain make, it should be up to the project developer's choice).

Also please note that the same tests are expected to be compiled with different toolchain versions, and possibly on different platforms (like native tests for portable projects).

[VGRSTM]

In sync. with @fred-r sounds Open-CMSIS is offering same as npm world somehow:

• npm install -g <package_name> <=> global install what sounds like $CMSIS_PACK_ROOT global repo usage used as reference by the project
• npm install <package_name> <=> local install what sounds like inside project local copy
• local package.json <=> cproject.yml file (@fred-r not csolution.yml because schema is not promoting components what's fine)
  As part of cproject.yml you can specify strict or looser version of each component like package.json

About pack size would say same if node world ... let's think about lodash (>3Mb despite you may need only one function out of ... )

[ilg-ul]

cproject.yml

I could not find a definition of this file in the Open-CMSIS-Pack 1.7.6 doxygen site, can you point me to the current proposal?

same if node world ... let's think about lodash (>3Mb ...

lodash is peanuts, ARM.CMSIS.5.8.0.pack is 36.2 MB, and Keil.STM32F4xx_DFP.2.15.0.pack is 260.8 MB.

[VGRSTM]

@ilg-ul schemas are here: https://github.com/Open-CMSIS-Pack/devtools/tree/main/tools/projmgr/schemas
(under dev. so part of tools dev. repo ... at that time)

lodash is peanuts but if you only require one function out of it ... it's toooo much :-) (was first package I was thinking about but I guess we may find some others ...)
if DFP I may agree toooo heavy :-) most of time .svd files and documentation are penalty I agree may be good to split here ...

[ilg-ul]

If you mean https://github.com/Open-CMSIS-Pack/devtools/blob/main/tools/projmgr/schemas/cproject.schema.json, which refers to a common.schema.json, I'm afraid I'm not smart enough to get it :-(

I would prefer a page intended for humans, with examples and even some explanations.

[VGRSTM]

A schema is a schema :-) main benefit about is such is up to date.

Such said maybe this YML Input Format is more helpful (cannot guarantee 100% up to date but so so bad anyway)

Test data may help too see https://github.com/Open-CMSIS-Pack/devtools/tree/main/tools/projmgr/test/data

[ilg-ul]

maybe this YML Input Format is more helpful

oops, lots of definitions...

Test data may help too see ...

Probably my question was confusing, since I tried to identify the syntax used for project dependencies, but neither the YML nor the test data files did not help me much :-(

---

I understand that you are trying to achieve very complex goals with your specs, and it seems that you try to define a unique YML format for everything. I'm a fan of layered design, so I'm not convinced that a one-size-fits-all approach is beneficial.

Back to my question, I'm looking for something that defines a list of dependencies. It should be a simple thing, for a given object (like a project, or a build configuration), it should enumerate a list of packages, with names and versions, that need to be downloaded from their original location (which is? where is the definition where the packages can be downloaded from? see #111) somewhere in a specific location on the local build machine.

I mean something like this:

• https://github.com/micro-os-plus/micro-test-plus-xpack/blob/db00b7f3e76aab3a04e9767449464c2deb77587e/package.json#L37-L41
• https://github.com/micro-os-plus/micro-test-plus-xpack/blob/db00b7f3e76aab3a04e9767449464c2deb77587e/package.json#L611-L621

Such definitions, processed by a tool (like xpm run install-all in this case) should be enough to bring in all desired packages, in order for the build(s) to be possible.

And possibly another helpful question to make myself clear: If I would like to add Open-CMSIS support to my packages, what would I use (in terms of configuration files syntax and tools) to get a similar functionality?

Of course, if you do not plan to open this for public use and keep it only for vendors (#111), the whole question no longer makes any sense.

[fred-r]

We cannot compare OpenCMSIS only with npm.
As you can see in the vanilla tooling proposed here, there are several aspects.

One is installing the packages (like npm).
But to me, npm is only "downloading" a list of libs.

OpenCMSIS does much more than this because there are configuration files, templates, generators.
And there is a project manager tool to handle the dependencies.

So packs are here to distribute SW, a bit like npm but via cpackget.
Then, on top of it, there are lots of versioning and configuration and generation concepts handled by the project manager (csolution).

Besides, OpenCMSIS aims at being open source, so at the end anybody can contribute an OpenCMSIS pack.
It is also possible to host it on any server.
If you want your packages to be OpenCMSIS compliant, in my opinion, you only need to provide a .pack, following the pdsc rules.
The YML files are project files created when using your packs to create a project.

[ilg-ul]

When CMSIS Packs came many years ago, I was very hopeful that I'll finally have a tool for my projects. After lots of efforts to support it in Eclipse, in the end I concluded that I could not use it; It was largely over-engineered and too MDK specific. :-(

I hoped that Open-CMSIS will clean things up, but it seems that things are getting even more complicated, and the separation between layers even more problematic.

The beauty of the npm public repository design is that it allows to publish packages with any content, which allowed me to use it for C/C++ source libraries as well.

"Downloading a list of libs" is the first layer in a mechanism to manage dependencies, and it can be independent of anything else.

---

So, for now at least, there is no clear answer on how to manage package dependencies. :-(

[fred-r]

In fact, I think OpenCMSIS gives you flexibility.

If you feel it is over-engineered for your needs then you can skip some possibilities : configuration files, templates, generators...etc...
You can very well declare packs with "basic" components providing source code or libs "as is".

Then, to express the dependencies, the granularity is the component, not the pack in my opinion.
The pack is only a way to convey software, what matters is the component.

Dependencies can be expressed via conditions in the pdsc.

But, ultimately, the pdsc should be seen (in my opinion) as the field of possibilities offered by the publisher of the pack.
The actual dependencies to be enforced are described in the project files (csoluton.yml and cproject.yml).
At this level, you can request an exact pack version and exact component versions (or ranges like in npm).

But, compared to npm I think the spirit here his more to have "software components" that you can "easily replace".
So, yes you can freeze everything to have very clear dependencies but you can also be more flexible to allow to replace a component by another component providing the same API.

I guess @ReinhardKeil and @jkrech can explain better than me :-)

[ilg-ul]

To summarise, the question was if OpenCMSIS provides a mechanism to automatically manage package dependencies.

The use case is an hypothetical GitHub project which uses several source libraries distributed as separate packages. The project is fully configured, the dependencies are known, all details are already in the project. The source libraries are not copied in the project, they are only mentioned as references, using a naming scheme and a version. On each push, a GitHub action is triggered, which needs to run some tests. I provided even links to a functional project already doing this.

The reason why I asked if Open-CMSIS Packs are intended to be project dependencies (or the content is still expected to be processed by a tool and the result copied into the project), was based on my previous experience with CMSIS Packs, since by the time I tried this, consuming CMSIS Packs was possible only via MDK, and there were not tools available for CI servers.

Up to now, after 6 replies, the question remains still unanswered, so I conclude that this is not one of the use cases of interest for Open-CMSIS, and it makes no sense to ask for more.

Feel free to label this topic as not relevant, or whatever labels you prefer.

[fred-r]

I am not sure to fully get the question.

Let me try this:

• say I have a npm package with dependencies
• if I install this package, its dependencies will be installed: when run without arguments, npm install downloads dependencies defined in a package.json file and generates a node_modules folder with the installed modules.

Now, with OpenCMSIS, it is slightly different:

• on one hand there are the CMSIS packs to distribute software components
• on the other hand there are projects using these components

So, with the tool 'csolution' you can create a csolution/cproject.yml file describing your project:

• the packs to be used
• the components to be used

Now, with the tool 'cpackget', you can install all required packs.
By default these packs will be installed in a local repo common to all projects referenced by $CMSIS_PACK_ROOT.

When using your project with csolution, the tool will look in $CMSIS_PACK_ROOT if the required packs and components are available.
If not, a missing dependency will be reported.
Then, with cpakcget you can download the missing pack(s).

OpenCMSIS delivers only the "atomic" tools.
Based on them you can write your own logic to determine the dependencies of your project and download the missing packs.

So, in your GitHub ACI situation:

1. you modify your project on your local machine to maybe update a dependency to take a newer version of pack Y already published on a CMSIS packs server
2. on your local machine, with cpackget you download this version of pack Y and test : you are happy with it
3. you push the update of the project (YML file) on github
4. a GitHub action is triggered to launch the ACI
   4a. on your server, csolution is invoked to determine the dependencies
   4b. on your server, cpackget is invoked to download the missing packs (on the server's disk)
   4a and 4b are iterated until all dependencies are resolved.
5. the ACI builds your project with csolution and cbuildgen
6. the ACI runs your tests

[ilg-ul]

you can write your own logic to determine the dependencies of your project and download the missing packs.

Ok, thank you. No problem if there is no simple and automatic solution, I can continue to use npm/xpm.

Good luck with your projects.

[fred-r]

Well, I believe the automatic solution will be provided by the tools implementing OpenCMSIS-Pack-spec.
OpenCMSIS-Pack-spec defines a spec.
The OpenCMSIS reference tooling is only for reference in my opinion, to be discussed with @ReinhardKeil and @jkrech.