Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(development): rewrite lock file implementation guide #31328

121 changes: 83 additions & 38 deletions docs/development/adding-a-package-manager.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,81 +111,126 @@

## Package files and Lock files

In Renovate terminology, "package files" are the files where human-readable dependency definitions are kept.
For example, this includes npm's `package.json` file, Maven's `pom.xml` file, and Docker's `Dockerfile`.
A special Renovate term is "package files".
Package files have human-readable dependency definitions, for example:

Some package managers may additionally have "lock files", e.g. npm's `package-lock.json`.
If a lock file is present in a repository then Renovate needs to update both in the same commit, otherwise the update may be "broken".
Therefore if a new manager is being developed and it is usual to have a lock file, supporting lock file updating should be done from the start.
- npm's `package.json` file
- Maven's `pom.xml` file
- Docker's `Dockerfile`

Supporting lock file updating usually requires Renovate to support a third party tool, e.g. `npm`, `poetry`, etc.
It's rare and not recommended for Renovate to "reverse engineer" lock file formats and make updates manually instead of calling such tools.
Adding support for such tools requires adding awareness of each tool to [Containerbase](https://github.com/containerbase/base) first.
Some package managers may also have "lock files", like npm's `package-lock.json`.

If a repository has a lock file, then Renovate MUST update the package file _and_ the matching lock file in the same commit.
rarkins marked this conversation as resolved.
Show resolved Hide resolved
This prevents creating "broken" updates for users and generating frustration with Renovate.

Check failure on line 124 in docs/development/adding-a-package-manager.md

View workflow job for this annotation

GitHub Actions / lint-docs

Trailing spaces

docs/development/adding-a-package-manager.md:124:92 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] https://github.com/DavidAnson/markdownlint/blob/v0.34.0/doc/md009.md
rarkins marked this conversation as resolved.
Show resolved Hide resolved

### Focus on getting lock file updates working
rarkins marked this conversation as resolved.
Show resolved Hide resolved

When you develop a new package manager, which supports lockfiles, focus on getting lock file updates working.
rarkins marked this conversation as resolved.
Show resolved Hide resolved
rarkins marked this conversation as resolved.
Show resolved Hide resolved

Supporting lock file updating usually means that Renovate runs a third-party tool, like `npm` or `poetry`.
rarkins marked this conversation as resolved.
Show resolved Hide resolved
The third-party tool then updates the lock file for Renovate.

### Avoid reverse engineering lock file formats

Let a third-party package manager (`npm`, `poetry`, etc.) update the lockfile.
Avoid "reverse engineering" lock file formats, where Renovate manually updates the lock file.

Only "reverse engineer" lock file formats as a last resort.

### Add third-party tools to Containerbase first

Add third-party package manager tools to [Containerbase](https://github.com/containerbase/base) first.

Here are the various ways in approximate order in which lock file awareness should be added to a manager:

### Lock file maintenance

The purpose of lock file maintenance is to update all locked dependencies (including transitive) to the latest possible versions.
The goal of lock file maintenance is to update all locked dependencies (including transitive dependencies) to the latest possible version.
HonkingGoose marked this conversation as resolved.
Show resolved Hide resolved

There are two ways to update lock files:

There are two approaches which can be used:
- Call a command like `<tool> update` to update all locked dependencies (plus transitive dependencies), updating the whole lock file where possible
- Delete the lock file, then call a command like `<tool> install` to create a new lockfile

- Delete the existing lock file, then call a command like `<tool> install` to regenerate it, or
- Call a command like `<tool> update` if such a command exists to satisfy this same requirement (updating the entire lock file where possible)
If you can, use the `<tool> update` method, as that keeps platform-specific information.

Where available, the second approach is better because lock file may sometimes have platform-specific information (e.g. amd64, arm64) which can be lost if the lock file is regenerated completely as in the first approach.
#### Keep platform-specific information

Lock files may have platform-specific information (e.g. `amd64`, `arm64`).
If you delete the lock file, and then create a new lock file with `<tool> install`, the platform-specific information is lost.

If you can, use the `<tool> update` method instead!

### Lock file updating after a package file change

This functionality is often mandatory from initial implementation.
Updating lock files after a package file changes is a fundamental feature.
This means you often need to build it first, when adding new package manager to Renovate.

In this scenario, an `updateArtifacts()` function must be added.
Its purpose is to essentially "sync" the lock file to the package file changes made by Renovate, so that both files can be updated in the same commit.
Add a `updateArtifacts()` function, that "syncs" the lock file to the package file changes made by Renovate.
This way, both files can be updated in the same commit.

Usually, the flow is like this:

1. Renovate makes changes to the version or constraint in the package file directly,
2. Renovate calls a tool command like "<tool> install", "<tool> lock", etc.
3. If the tool command resulted in a changed lock file (it usually should), then Renovate commits the changes along with the package file change
1. Renovate directly changes the version or constraint in the package file,
2. Renovate calls a tool command like `<tool> install`, `<tool> lock`, etc.
3. If the tool command changes the lock file (which it usually should!), then Renovate commits the changed lock file and the package file

### Locked version extracting and dependency pinning

The next step is for the manager's "extract" functionality to return a `lockedVersion` for dependencies whenever a lock file exists.
To do this, the manager should:
The next step is for the manager's "extract" feature to return a `lockedVersion` for dependencies, whenever a lock file exists.
To do this, the manager must:
rarkins marked this conversation as resolved.
Show resolved Hide resolved
rarkins marked this conversation as resolved.
Show resolved Hide resolved

1. Parse the lock file
2. Associate each dependency from the package file with its entry in the lock file
3. Add that associated version as `lockedVersion`
2. Match each dependency from the package file to its entry in the lock file
3. Add the matched version as `lockedVersion`

Once `lockedVersion` is provided, Renovate should be able to "pin" constraints/ranges into exact versions, if the user configures as such (e.g. `rangeStrategy=pin`) however Renovate _won't_ automatically be able to make lockfile-only updates.
Once `lockedVersion` is provided, Renovate should be able to "pin" constraints/ranges into exact versions, if the user configures as such (e.g. `rangeStrategy=pin`), however Renovate _won't_ automatically be able to make lockfile-only updates.

### Lock file-only updates

#### updateArtifacts()

It's a common scenario where users want or need to retain constraints in their package file (e.g. `^1.0.0`) and have Renovate make updates to the lock file when new versions are available (e.g. updating from a locked value of `1.1.0` to `1.1.1`).
In this case, it's a prerequisite that the manager must extract `lockedVersion` as described above.
End users often want, or need to:

- preserve constraints in their package file (like `^1.0.0`)
- and want Renovate to update their lock file when a new versions is available (e.g. updating from a locked value of `1.1.0` to `1.1.1`)

In this case, the Renovate package manager must extract the `lockedVersion` as described above!

#### Detect if updates satisfy `isLockFileUpdate=true`

In addition to this, the manager needs to add logic to `updateArtifacts()` to detect if _any_ of the updates it has been passed satisfy `isLockFileUpdate=true`.

In addition to this, the manager needs to add logic to `updateArtifacts()` to detect if any of the updates it has been passed satisfy `isLockFileUpdate=true`.
If any lock file-only updates have been passed, then the manager typically needs to run specific commands to update/bump the locked version for one specific dependency only.
This functionality is manager-specific, and depends heavily on the capabilities of the third party tool, but a mix of the following approaches are used in Renovate, from best to worst:
This functionality is manager-specific, and depends heavily on the capabilities of the third-party tool.
A mix of the following approaches are used in Renovate, from best to worst:

- Renovate calls a tool command to specifically update the dependency in question to the specific version, e.g. `<tool> update <dependency name>@<new version>`
- Renovate manually updates the locked version in the lock file it needs updated, then calls a `<tool> install` command to "fix" up the remaining parts (hashes, transitive dependencies, etc). This is good if it works but it is prone to breaking in future releases because it's possible that the maintainers of the tool are not aware of people using it in this manner, even if it works unintentionally.
- Renovate calls a tool command similar to the first approach, except the tool doesn't support specific versions, e.g. `<tool> update <dependency name>`. This approach can be problematic because Renovate might _want_ to update to e.g. v1.1.1 but instead the tool finds a newer v1.1.2 and that's what the user gets instead
- Renovate manually updates the locked version in the lock file it needs updated, then calls a `<tool> install` command to "fix" up the remaining parts (hashes, transitive dependencies, etc). This is good, if it works, but it is prone to breaking in future releases if the maintainers of the tool do not know people are using their package manager in this manner, even if it works unintentionally.
- Renovate calls a tool command similar to the first approach, except the tool does not support specific versions, e.g. `<tool> update <dependency name>`. This approach can be problematic because Renovate might _want_ to update to e.g. `v1.1.1` but instead the tool finds a newer `v1.1.2` and that's what the user gets instead

##### Difficult cases

A further complication is that sometimes:

A further complication is that sometimes dependencies need to be upgraded together or else there are peer dependency problems or other conflicts.
In that case it's best if the tool can support a list of dependencies to update and they are done all at once.
- dependencies must be upgraded together
- there are peer dependency problems
- or there is some other conflict

In those cases it's best if the tool supports creating a list of dependencies to update, and the tool then updates all dependencies at once.

#### updateLockedDependency()

The `updateLockedDependency()` method is optional for managers but recommended that any manager which supports `rangeStrategy=update-lockfile` implements the `updateLockedDependency()` method.
The most valuable part of this method is returning quickly if a dependency is already updated, so that tool commands don't need to be run every time.
The `updateLockedDependency()` method is optional for managers.
But we recommend that any manager which supports `rangeStrategy=update-lockfile` also implements the `updateLockedDependency()` method.

The goal of the `updateLockedDependency()` method is to return quickly if a dependency is already updated.
This way, Renovate only runs tool commands when there is a dependency to update.

The simplest logic for this method is:
The simplest logic for `updateLockedDependency()` is:

1. Parse the existing lock file
2. If the locked version of the dependency is already updated to the version specified then return `{ status: 'already-updated' }`
3. Otherwise, return `{ status: 'unsupported' }`
2. If the locked version of the dependency is already updated to the version specified: return `{ status: 'already-updated' }`
3. Else: return `{ status: 'unsupported' }`

An example of this can be seen in [the composer manager source code for updateLockedDependency()](https://github.com/renovatebot/renovate/blob/da4964ac05952f9fe0543ba1174fcd62ad083d48/lib/modules/manager/composer/update-locked.ts#L7-L30).=
An example of this can be seen in [the `composer` manager source code for `updateLockedDependency()`](https://github.com/renovatebot/renovate/blob/da4964ac05952f9fe0543ba1174fcd62ad083d48/lib/modules/manager/composer/update-locked.ts#L7-L30).
Loading