purescript-contrib spago@next migration

[jamesdbrock]

@srghma has submitted PRs for migrating all the repos to spago@next and here is a place for discussion about how that should be done. I want to produce a “spago next migration guide” in this discussion, so that I understand how this should be done.

(I've turned on Github Discussions for purescript-contrib. We could also have this conversation on discourse.purescript.org)

I think we should either keep the bowerfiles and expect contrib libraries to keep the spago.yaml and bower.json files in sync, or remove the bowerfiles altogether at this point. I don't see why we would keep a bowerfile but remove all the dependencies from it.

purescript-contrib/purescript-profunctor-lenses#148 (comment)

I would prefer if we keep the bower files for now, unless we're going to make all these updates breaking releases too, as I think it'll break existing projects that use bower otherwise. Like my main work project.

purescript-contrib/purescript-profunctor-lenses#148 (comment)

For a library, the spago.yaml file needs to specify version ranges and should not use a package set. For example, this should be:

- arrays: >=7.0.0 <8.0.0

for parity with the bower.json file. It won't error right away but this is necessary to publish the package to the registry and for other libraries to depend on this library.

purescript-contrib/purescript-profunctor-lenses#148 (comment)

I agree with making these breaking changes in any case.

purescript-contrib/purescript-profunctor-lenses#148 (comment)

If you migrate to spago@next then a few things come to mind:

• you may want to avoid committing lockfiles for the libraries, especially because the format is unstable and likely to change while spago is unstable.
• you will need to make sure you add version ranges for dependencies and don't use a package set; contrib repos currently have bower.json and spago.dhall files, but the spago.dhall is just internal and for convenience, it's irrelevant for publishing. the new spago.yaml files are relevant for library publishing and will need accurate version bounds.

https://discord.com/channels/864614189094928394/938253816862736405/1293989090588627087

Personally I think we should check spago.lock into the repo? When the spago.lock file format has changed, this is what happens:

Which is fine.

Please let's release all these updated libraries with a new major versions to prevent breakages with existing bower projects.

https://github.com/orgs/purescript-contrib/discussions/1#discussioncomment-10930739

Migration Guide Draft

• spago.yaml should specify ranges and should not use a package set. Bump major version of the package.
• spago.lock check it in.
• bower.json delete.
• packages.json delete.
• psc_package.json delete.
• spago.dhall delete.
• packages.dhall delete.

[srghma]

Migration Guide Draft

ok, this is for me

spago.yaml should specify ranges

already done

and should not use a package set.

yes, done, registry: 60.5.0

bower.json delete.

ok

will move the "authors" field from bower.json to then end of README.md and remove bower.json from all projects

packages.json delete.
spago.dhall delete.

should already be done

[srghma]

then end of README.md

no, to authors in packages.json

like in purescript-ace/package.json

{
  "private": true,
  "license": "MIT",
  "authors": [
    "John A. De Goes <[email protected]> (http://degoes.net)",
    "Maxim Zimaliev <[email protected]>"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/purescript-contrib/purescript-ace.git"
  },
  "name": "purescript-ace",
  "scripts": {
    "build": "eslint src && spago build --censor-stats --strict --ensure-ranges --pedantic-packages",
    "test": "spago test --offline"
  },
  "devDependencies": {
    "eslint": "^9.12.0"
  },
  "dependencies": {
    "ace-builds": "^1.36.2"
  }
}

N.B. this field doesnt do anything , unlike https://docs.npmjs.com/cli/v8/configuring-npm/package-json#people-fields-author-contributors

[srghma]

done

[garyb]

Please let's release all these updated libraries with a new major versions to prevent breakages with existing bower projects.

[srghma]

const fs = require('fs');
const path = require('path');
const axios = require('axios');
const YAML = require('yaml');
const { execSync } = require('child_process');

// Set the output directory
const outputDir = path.join(process.env.HOME, 'projects', 'contrib');

// Function to get the latest version of a package from Pursuit
async function getLatestVersion(packageName) {
  try {
    const url = `https://pursuit.purescript.org/packages/purescript-${packageName}`;
    const response = await axios.get(url);

    // The final URL after redirection
    const redirectedUrl = response.request.res.responseUrl;

    // Extract the version from the URL using regex
    const versionMatch = redirectedUrl.match(/\/(\d+\.\d+\.\d+)$/);

    if (versionMatch) {
      const latestVersion = versionMatch[1];
      console.log(`Latest version: ${latestVersion}`);
      return latestVersion;
    } else {
      throw new Error('Could not find version in the URL');
    }
  } catch (error) {
    console.log('error.response.status', error.response.status, packageName)
    if (error.response && error.response.status === 404) {
      return null
    } else {
      console.error('Failed to fetch package information.');
      return null
    }
  }
}

var directories = fs.readdirSync(outputDir).filter(x => x !== 'chnglg').map(dir => {
  const pathToSpagoYaml = path.join(outputDir, dir, 'spago.yaml')
  const spagoConfig = YAML.parse(fs.readFileSync(pathToSpagoYaml, 'utf8'));
  const packageVersion = spagoConfig.package.publish.version;
  const packageName = spagoConfig.package.name;
  return { spagoConfig, dir, packageVersion, packageName, pathToSpagoYaml }
});

var packages_ = await Promise.all(directories.map(async (config) => {
  const { packageName } = config
  const latestVersion = await getLatestVersion(packageName);
  return { ...config, latestVersion }
}));

var packages__ = packages_.map((config) => {
  const { spagoConfig, dir, packageVersion, packageName, pathToSpagoYaml, latestVersion } = config
  const versionParts = packageVersion.split('.');

  let output;
  if (latestVersion) {
    // Perform the major bump logic
    versionParts[0] = (parseInt(versionParts[0]) + 1).toString(); // Major bump
    versionParts[1] = '0'; // Reset minor version
    versionParts[2] = '0'; // Reset patch version
    output = versionParts.join('.');
    console.log(`Package version should be major bumped from ${packageVersion} to ${output} ${packageName}`);

    const spagoYamlPath = path.join(outputDir, dir, 'spago.yaml');
    delete spagoConfig.version
    spagoConfig.package.publish.version = output;
    const newYamlContent = YAML.stringify(spagoConfig);
    fs.writeFileSync(spagoYamlPath, newYamlContent, 'utf8');

    // Navigate to the package directory and commit changes
    try {
      // execSync(`git diff`, { cwd: path.join(outputDir, dir) });
      execSync(`git add spago.yaml`, { cwd: path.join(outputDir, dir) });
      execSync(`git commit -m "feat: bump version to ${output}" --amend`, { cwd: path.join(outputDir, dir) });
      execSync(`git push --force`, { cwd: path.join(outputDir, dir) });
    } catch (error) {
      console.error('Error committing changes to Git:', error.message);
    }
  } else {
    output = packageVersion; // Leave version as is
    console.log(`Package is not yet published; version remains as ${output} ${packageName}`);
  }
})

Package version should be major bumped from 9.1.0 to 10.0.0 ace
Package version should be major bumped from 8.0.0 to 9.0.0 aff
Package version should be major bumped from 6.0.0 to 7.0.0 aff-bus
Package version should be major bumped from 9.0.0 to 10.0.0 aff-coroutines
Package version should be major bumped from 13.0.0 to 14.0.0 affjax
Package version should be major bumped from 1.0.0 to 2.0.0 affjax-node
Package version should be major bumped from 1.0.0 to 2.0.0 affjax-web
Package version should be major bumped from 9.0.0 to 10.0.0 argonaut
Package version should be major bumped from 9.1.0 to 10.0.0 argonaut-codecs
Package version should be major bumped from 7.0.0 to 8.0.0 argonaut-core
Package version should be major bumped from 8.0.0 to 9.0.0 argonaut-generic
Package version should be major bumped from 10.0.0 to 11.0.0 argonaut-traversals
Package version should be major bumped from 13.2.0 to 14.0.0 arraybuffer
Package version should be major bumped from 3.0.2 to 4.0.0 arraybuffer-types
Package version should be major bumped from 5.0.0 to 6.0.0 avar
Package version should be major bumped from 7.0.1 to 8.0.0 bigints
Package version should be major bumped from 7.0.1 to 8.0.0 colors
Package version should be major bumped from 3.0.0 to 4.0.0 concurrent-queues
Package version should be major bumped from 7.0.0 to 8.0.0 coroutines
Package version should be major bumped from 6.0.0 to 7.0.0 css
Package version should be major bumped from 7.0.0 to 8.0.0 fixed-points
Package version should be major bumped from 2.0.0 to 3.0.0 float32
Package version should be major bumped from 6.0.0 to 7.0.0 fork
Package version should be major bumped from 7.0.0 to 8.0.0 form-urlencoded
Package version should be major bumped from 7.0.0 to 8.0.0 formatters
Package version should be major bumped from 7.0.0 to 8.0.0 freet
Package version should be major bumped from 0.5.0 to 1.0.0 github-actions-toolkit
Package version should be major bumped from 6.0.0 to 7.0.0 http-methods
Package version should be major bumped from 3.0.0 to 4.0.0 int64
Package version should be major bumped from 1.0.0 to 2.0.0 js-abort-controller
Package version should be major bumped from 2.2.1 to 3.0.0 js-bigints
Package is not yet published; version remains as 1.0.0 js-blob
Package version should be major bumped from 8.0.0 to 9.0.0 js-date
Package version should be major bumped from 0.2.1 to 1.0.0 js-fetch
Package version should be major bumped from 1.0.0 to 2.0.0 js-promise
Package version should be major bumped from 4.0.0 to 5.0.0 js-promise-aff
Package version should be major bumped from 6.1.0 to 7.0.0 js-timers
Package version should be major bumped from 3.1.0 to 4.0.0 js-uri
Package version should be major bumped from 7.0.0 to 8.0.0 machines
Package version should be major bumped from 1.0.0 to 2.0.0 matryoshka
Package version should be major bumped from 6.0.0 to 7.0.0 media-types
Package version should be major bumped from 6.0.0 to 7.0.0 now
Package version should be major bumped from 6.0.0 to 7.0.0 nullable
Package version should be major bumped from 7.0.0 to 8.0.0 options
Package version should be major bumped from 6.0.0 to 7.0.0 optparse
Package version should be major bumped from 6.0.0 to 7.0.0 precise
Package version should be major bumped from 8.0.0 to 9.0.0 profunctor-lenses
Package version should be major bumped from 7.0.0 to 8.0.0 quickcheck-laws
Package version should be major bumped from 6.0.0 to 7.0.0 rationals
Package version should be major bumped from 11.0.0 to 12.0.0 react
Package version should be major bumped from 8.0.0 to 9.0.0 react-dom
Package version should be major bumped from 11.0.0 to 12.0.0 routing
Package version should be major bumped from 8.0.0 to 9.0.0 string-parsers
Package version should be major bumped from 4.0.0 to 5.0.0 strings-extra
Package version should be major bumped from 6.0.0 to 7.0.0 these
Package version should be major bumped from 7.0.0 to 8.0.0 uint
Package version should be major bumped from 6.0.0 to 7.0.0 unicode
Package version should be major bumped from 5.0.0 to 6.0.0 unsafe-reference
Package version should be major bumped from 9.0.0 to 10.0.0 uri

[thomashoneyman]

I know it’s annoying, but we really ought to be careful about these breaking releases and the version ranges we assign to these packages. If a package like aff is at 7.1.0 and is updated to 8.0.0, then a package that depends on it really should support the range >=7.10 <9.0.0 (instead of the typical one major version >=8.0.0 <9.0.0) because the release is not breaking as far as the spago consumers are concerned. We’re just trying to avoid breaking Bower projects.

A second consideration we’ve always taken when doing breaking releases is to pause and go through the issues and PRs for the affected libraries to see if there are breaking changes we can go ahead and include. We do them infrequently so if a breaking change doesn’t get merged now it could quite easily have to wait a long time before it gets in. I don’t think there are many of these sitting around but it’s worth considering.

I’m unfortunately stretched for time and can’t help out with the updates, so I guess it’s not really my call whether to include breaking changes wherever possible or not, but I wanted to bring it up. That’s how Jordan and I used to handle these and I think it worked well.

[srghma]

We do them infrequently so if a breaking change doesn’t get merged now it could quite easily have to wait a long time before it gets in

I do have breaking changes prs - purescript-contrib/purescript-pathy#52 + purescript-contrib/purescript-pathy#54

and in purescript-node purescript-node/purescript-node-fs#86

[jamesdbrock]

Please let's release all these updated libraries with a new major versions to prevent breakages with existing bower projects.

Option 2 seems better to me? What bad things can happen if we do option 2, keep bower.json and bump patch version?

1. Delete bower.json, bump major version

If we delete the bower.json, then we should bump the major version to signal that if bower projects upgrade this dependency then they will break.

2. Keep bower.json, bump patch version

If we keep the bower.json, then can we just bump the patch version?

This is nice because then dependent projects can just upgrade to the patch version without bumping their own major version and causing “breaking change” major version bumps all the way up the dependency chain.

We can then delete the bower.json in the future.

[jamesdbrock]

I know you already did option 1 and pushed it. Before we merge your PRs I want to be sure that it's not going to cause problems. Thank you for doing all this work. I'm trying to ensure that your work isn't wasted.

[thomashoneyman]

If we preserve the bower.json files, then we must keep them in sync with the spago.yaml files, and we can do patch releases. Historically, keeping two manifests in sync hasn't gone well; one or the other generally falls out of date (usually the bower file).

If we don't preserve them, then we don't have to keep two manifests in sync, but we must do breaking releases. That gives us an opportunity to sweep through and do other breaking changes together though.

So there are tradeoffs to each of these. @garyb do you think we should do a Discourse discussion about this? We often do that when making a decision that will affect a lot of people, like dropping Bower support.

[thomashoneyman]

Also cc: @f-f.

[f-f]

We should be sticking to one way of tracking version bounds. Having both the bower.json and the spago.yaml means that there are two places to store them, which will produce different build plans (even if the bounds are the same the resolution algorithms are different), so now the registry has to pick one to favour for its build plans, and downstream libraries might get confused about their own bounds.

So I'd recommend to either:

1. move to spago.yaml and remove the bower files, or
2. not move to spago.yaml yet

We'll have to do (1) in the near future anyways, (2) would just delay the whole thing.

[garyb]

So there are tradeoffs to each of these. @garyb do you think we should do a Discourse discussion about this? We often do that when making a decision that will affect a lot of people, like dropping Bower support.

Yeah, we probably should do that. I don't really know how many people are using the bower method still, I wouldn't be surprised if nobody apart from me cares about it at this point. 😉 It's been quite a while since package sets were introduced and anyone coming to PS fresh would probably find bower's... let's be charitable and say "quirks"... unpalatable.

The "...psst! While Bower is maintained, we recommend using <...> for front-end projects." note has been up since 2017 at this point, which I remember scared people off for a while since I think that's pre-spago even so there wasn't an alternative at the time.