vega/vega-lite version problem

[HaveF]

The Vega/Lite JS file is embedded in the repository, making it difficult to update easily. Perhaps we can consider setting up a cron job in GitHub Actions to automate the updating process? Or just set a package.json for renovate notification?

The reason for exploring this solution is because while debugging the graph today, I noticed some discrepancies between opening the Vega Editor via datasette-dashboards and viewing it in the dashboards.

https://github.com/vega/vega/releases
5.25.0
https://github.com/vega/vega-lite/releases
5.14.1
https://github.com/vega/vega-embed/releases
6.22.2

Our version maybe is this:

[email protected]
[email protected]
[email protected]

[rclement]

Hi @HaveF,

Indeed I embedded external JS/CSS dependencies within datasette-dashboards to avoid external CDN calls (and protect privacy when using the plugin in secure contexts).

I have been using extensively the Node.js ecosystem in the past, and I'd rather not include it with a Python project... Unless you can find a solution to the following:

• Define dependencies in a minimal package.json file:

{
  "name": "datasette-dashboard",
  "version": "0.6.2",
  "description": "Datasette plugin providing data dashboards from metadata",
  "dependencies": {
    "vega": "^5.25.0",
    "vega-embed": "^6.22.2",
    "vega-lite": "^5.15.0"
  }
}

• Allow to vendor dependencies in datasette_dashboards/static/vendor when running npm install (instead of node_modules)
  • Apparently there is npm i --prefix datasette_dashboards/static/vendor
• Only include built JS/CSS artifacts so that we obtain (currently runing npm i includes all transitive dependencies + source code etc.):

datasette_dashboards/static/vendor
|-- vega.js
|-- vega-embed.js
|-- vega-lite.js 

I did not manage to achieve this but I'm open to a clean solution relying on package.json so that renovate can notify when new versions are availables

[HaveF]

Indeed I embedded external JS/CSS dependencies within datasette-dashboards to avoid external CDN calls (and protect privacy when using the plugin in secure contexts).

I really appreciate this idea of yours ❤️

[rclement]

Unfortunately, this is one of the hotest debates with CDN usages for professional applications

[rclement]

I just found out that Datasette has the same "problem" for CodeMirror and here is the implemented solution:
https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror

The trick is to install node modules in a standard way (using the node_modules directory) and then to bundle selected librairies using rollup. For instance for vega:

./node_modules/.bin/rollup node_modules/vega/build/vega.min.js --file datasette_dashboards/static/vega.min.js

I quickly tried it it seems to be working. And combined with a "postinstall" script performing the bundling in package.json it might be a nice solution simply using npm install

[rclement]

Here is an experimental PR with the above workflow: #176
Updating external JS dependencies is simply a matter of updating the package.json file and run npm install that will bundle vendored JS files into datasette-dashboards static assets

[HaveF]

the post upgrade tasks can be curl -o datasette_dashboards/static/vega-embed.min.js https://cdn.jsdelivr.net/npm/vega-embed@latest

[rclement]

Wait... when and where does the command npm install run? On our local machine? I suppose it should be the job of renovate, right?

Either Renovate or a developer of datasette-dashboards (me) when anticipating Renovate. But this is an ad-hoc command in the workflow, separate from the Python workflow. A user of datasette-dashboards never has to do it!

It seems like use postupgradetasks before a commit is made by Renovate, right?

The thing is that there is no need for this hook, as the "postinstall" script is already called at the end of npm install by Renovate. Besides, not a fan of introducing a logic binding within the Renovate config, I like to use standard workflows

[rclement]

Honestly, thanks for bringing this up (it was somewhere in my mind but never took the time to reflect on it) but I do like the state of the PR #176 right now:

• It is using a minimal but standard package.json + npm workflow
• Static assets for datasette-dashboards are installed automatically by npm install
• It is compatible with Renovate for automatic deps update
• This is transparent for users of the plugin, same as before

[HaveF]

• It is using a minimal but standard package.json + npm workflow

That's fine. I'm comfortable with this too 😄

[HaveF]

Thanks for your work! ❤️

[rclement]

JS dependencies are now tracked with package.json (no need for a lock file in this case) and Renovate triggers PRs when new versions are available.

A few caveats:

• By default, Renovate is skipping NPM package installation when resolving new versions
• Enabling skipInstall and/or postUpgradeTasks is only possible with a self-hosted Renovate instance, not Renovate cloud
• To perform the bundling of JS assets, a GitHub Action dispatch workflow can be triggered manually on PRs opened by Renovate

[HaveF]

Thanks for your caveats!