Skip to content

Commit

Permalink
Merge pull request #3501 from samansmink/add-extension-versioning
Browse files Browse the repository at this point in the history 
Add extension versioning
  • Loading branch information
@szarnyasg
szarnyasg authored Sep 9, 2024
2 parents 9dea69d + 6d1c1c1 commit 20f5f20
Show file tree
Hide file tree
Showing 2 changed files with 114 additions and 32 deletions.
117 changes: 93 additions & 24 deletions docs/extensions/versioning_of_extensions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,39 +5,108 @@ title: Versioning of Extensions

## Extension Versioning

Just like DuckDB itself, DuckDB extensions have a version. This version can be used by users to determine which features are available
in the extension they have installed, and by developers to understand bug reports. DuckDB extensions can be versioned in different ways:
Most software has some sort of version number. Version numbers serve a few important goals:

**Extensions whose source lives in DuckDB's main repository** (in-tree extensions) are tagged with the short git hash of the repository.
For example, the parquet extension is built into DuckDB version `v0.10.3` (which has commit `70fd6a8a24`):
* Tie a binary to a specific state of the source code
* Allow determining the expected feature set
* Allow determining the state of the APIs
* Allow efficient processing of bug reports (e.g., bug `#1337` was introduced in version `v3.4.5` )
* Allow determining chronological order of releases (e.g., version `v1.2.3` is older than `v1.2.4`)
* Give an indication of expected stability (e.g., `v0.0.1` is likely not very stable, whereas `v13.11.0` probably is stable)

```sql
SELECT extension_name, extension_version, install_mode
FROM duckdb_extensions()
WHERE extension_name='parquet';
```
Just like [DuckDB itself]({% link docs/dev/release_calendar.md %}), DuckDB extensions have their own version number. To ensure consistent semantics
of these version numbers across the various extensions, DuckDB's [Core Extensions]({% link docs/extensions/core_extensions.md %}) use
a versioning scheme that prescribes how extensions should be versioned. The versioning scheme for Core Extensions is made up of 3 different stability levels: **unstable**, **pre-release**, and **stable**.
Let's go over each of the 3 levels and describe their format:

<div class="narrow_table"></div>
### Unstable Extensions

| extension_name | extension_version | install_mode |
|:------------------|:------------------|:---------------------|
| parquet | 70fd6a8a24 | STATICALLY_LINKED |
Unstable extensions are extensions that can't (or don't want to) give any guarantees regarding their current stability,
or their goals of becoming stable. Unstable extensions are tagged with the **short git hash** of the extension.

**Extensions whose source lives in a separate repository** (out-of-tree extensions) have their own version. This version is **either**
the short git hash of the separate repository, **or** the git version tag in [Semantic Versioning](https://semver.org/) format.
For example, in DuckDB version `v0.10.3`, the azure extension could be versioned as follows:
For example, at the time of writing this, the version of the `vss` extension is an unstable extension of version `690bfc5`.

```sql
SELECT extension_name, extension_version, install_mode
FROM duckdb_extensions()
WHERE extension_name = 'azure';
What to expect from an extension that has a version number in the **unstable** format?

* The state of the source code of the extension can be found by looking up the hash in the extension repository
* Functionality may change or be removed completely with every release
* This extension's API could change with every release
* This extension may not follow a structured release cycle, new (breaking) versions can be pushed at any time

### Pre-Release Extensions

Pre-release extensions are the next step up from Unstable extensions. They are tagged with version in the **[SemVer](https://semver.org/)** format, more specifically, those in the `v0.y.z` format.
In semantic versioning, versions starting with `v0` have a special meaning: they indicate that the more strict semantics of regular (`>v1.0.0`) versions do not yet apply. It basically means that an extensions is working
towards becoming a stable extension, but is not quite there yet.

For example, at the time of writing this, the version of the `delta` extension is a pre-release extension of version `v0.1.0`.

What to expect from an extension that has a version number in the **pre-release** format?

* The extension is compiled from the source code corresponding to the tag.
* Semantic Versioning semantics apply. See the [Semantic Versioning](https://semver.org/) specification for details.
* The extension follows a release cycle where new features are tested in nightly builds before being grouped into a release and pushed to the `core` repository.
* Release notes describing what has been added each release should be available to make it easy to understand the difference between versions.

### Stable Extensions

Stable extensions are the final step of extension stability. This is denoted by using a **stable SemVer** of format `vx.y.z` where `x>0`.

For example, at the time of writing this, the version of the `parquet` extension is a stable extension of version `v1.0.0`.

What to expect from an extension that has a version number in the **stable** format? Essentially the same as pre-release extensions, but now the more
strict SemVer semantics apply: the API of the extension should now be stable and will only change in backwards incompatible ways when the major version is bumped.
See the SemVer specification for details

## Release Cycle of Pre-Release and Stable Core Extensions

In general for extensions the release cycle depends on their stability level. **unstable** extensions are often in
sync with DuckDB's release cycle, but may also be quietly updated between DuckDB releases. **pre-release** and **stable**
extensions follow their own release cycle. These may or may not coincide with DuckDB releases. To find out more about the release cycle of a specific
extension, refer to the documentation or GitHub page of the respective extension. Generally, **pre-release** and **stable** extensions will document
their releases as GitHub releases, an example of which you can see in the [delta extension](https://github.com/duckdb/duckdb_delta/releases).

Finally, there is a small exception: All [in-tree]({% link docs/extensions/working_with_extensions.md %}#in-tree-vs-out-of-tree) extensions simply
follow DuckDB's release cycle.

## Nightly Builds

Just like DuckDB itself, DuckDB's core extensions have nightly or dev builds that can be used to try out features before they are officially released. This
can be useful when your workflow depends on a new feature, or when you need to confirm that your stack is compatible with the upcoming version.

Nightly builds for extensions are slightly complicated due to the fact that currently DuckDB extensions binaries are tightly bound to a single DuckDB version. Because of this tight connection,
there is a potential risk for a combinatory explosion. Therefore, not all combinations of nightly extension build and nightly DuckDB build are available.

In general, there are 2 ways of using nightly builds: using a nightly DuckDB build and using a stable DuckDB build. Let's go over the differences between the two:

### From Stable DuckDB

In most cases, user's will be interested in a nightly build of a specific extension, but don't necessarily want to switch to using the nightly build of DuckDB itself. This allows using a specific bleeding-edge
feature while limiting the exposure to unstable code.

To achieve this, Core Extensions tend to regularly push builds to the [`core_nightly` repository]({% link docs/extensions/working_with_extensions.md %}#extension-repositories). Let's look at an example:

First we install a [**stable DuckDB build**]({% link docs/installation/index.html %}).

Then we can install and load a **nightly** extension like this:

```bash
INSTALL aws FROM core_nightly;
LOAD aws;
```

<div class="narrow_table"></div>
In this example we are using the latest **nightly** build of the aws extension with the latest **stable** version of DuckDB.

### From Nightly DuckDB

| extension_name | extension_version | install_mode |
|:---------------|:------------------|:---------------|
| azure | 49b63dc | REPOSITORY |
When DuckDB CI produces a nightly binary of DuckDB itself, the binaries are distributed with a set of extensions that are pinned at a specific version. This extension version will be tested for that specific build of DuckDB, but might not be the latest dev build. Let's look at an example:

First, we install a [**nightly DuckDB build**]({% link docs/installation/index.html %}). Then, we can install and load the `aws` extension as expected:

```sql
INSTALL aws;
LOAD aws;
```

## Updating Extensions

Expand Down
29 changes: 21 additions & 8 deletions docs/extensions/working_with_extensions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ Some extensions are distributed for the following platforms:
* `windows_amd64_rtools`
* `wasm_eh` and `wasm_mvp` (see [DuckDB-Wasm's extensions]({% link docs/api/wasm/extensions.md %}))

For platforms outside the ones listed above, we do not officially distribute extensions (e.g., `linux_arm64_gcc4`, `windows_amd64_mingw`).
For platforms outside the ones listed above, we do not officially distribute extensions (e.g., `linux_arm64_android`, `linux_arm64_gcc4`, `windows_amd64_mingw`).

### Sharing Extensions between Clients

The shared installation location allows extensions to be shared between the client APIs _of the same DuckDB version_, as long as they share the same `platfrom` or ABI. For example, if an extension is installed with version 0.10.0 of the CLI client on macOS, it is available from the Python, R, etc. client libraries provided that they have access to the user's home directory and use DuckDB version 0.10.0.
The shared installation location allows extensions to be shared between the client APIs _of the same DuckDB version_, as long as they share the same `platform` or ABI. For example, if an extension is installed with version 0.10.0 of the CLI client on macOS, it is available from the Python, R, etc. client libraries provided that they have access to the user's home directory and use DuckDB version 0.10.0.

## Extension Repositories

Expand Down Expand Up @@ -78,7 +78,7 @@ INSTALL spatial FROM 'http://nightly-extensions.duckdb.org';
To install an extensions from a custom repository unknown to DuckDB:

```sql
INSTALL custom_extension FROM 'https://my-custom-extension-repository';
INSTALL ⟨custom_extension⟩ FROM 'https://my-custom-extension-repository';
```

Alternatively, the `custom_extension_repository` setting can be used to change the default repository used by DuckDB:
Expand All @@ -87,11 +87,11 @@ Alternatively, the `custom_extension_repository` setting can be used to change t
SET custom_extension_repository = 'http://nightly-extensions.duckdb.org';
```

While any url or local path can be used as a repository, currently DuckDB contains the following predefined repositories:
While any URL or local path can be used as a repository, DuckDB currently contains the following predefined repositories:

<div class="narrow_table"></div>
<div class="narrow_tabl"></div>

| Alias | Url | Description |
| Alias | URL | Description |
|:----------------------|:---------------------------------------|:---------------------------------------------------------------------------------------|
| `core` | `http://extensions.duckdb.org` | DuckDB core extensions |
| `core_nightly` | `http://nightly-extensions.duckdb.org` | Nightly builds for `core` |
Expand All @@ -109,7 +109,9 @@ INSTALL aws FROM core_nightly;
SELECT extension_name, extension_version, installed_from, install_mode FROM duckdb_extensions();
```

Would output:
This outputs:

<div class="narrow_table monospace_table"></div>

| extensions_name | extensions_version | installed_from | install_mode |
|:----------------|:-------------------|:---------------|:-------------|
Expand All @@ -120,7 +122,7 @@ Would output:
### Creating a Custom Repository

A DuckDB repository is an HTTP, HTTPS, S3, or local file based directory that serves the extensions files in a specific structure.
This structure is describe [here](#downloading-extensions-directly-from-s3), and is the same
This structure is described in the [“Downloading Extensions Directly from S3” section](#downloading-extensions-directly-from-s3), and is the same
for local paths and remote servers, for example:

```text
Expand Down Expand Up @@ -225,6 +227,17 @@ For building and installing extensions from source, see the [building guide]({%

To statically link extensions, follow the [developer documentation's “Using extension config files” section](https://github.com/duckdb/duckdb/blob/main/extension/README.md#using-extension-config-files).

## In-Tree vs. Out-of-Tree

Originally, DuckDB extensions lived exclusively in the DuckDB main repository, `github.com/duckdb/duckdb`. These extensions are called in-tree. Later, the concept
of out-of-tree extensions was added, where extensions where separated into their own repository, which we call out-of-tree.

While from a user's perspective, there are generally no noticeable differences, there are some minor differences related to versioning:

* in-tree extensions use the version of DuckDB instead of having their own version
* in-tree extensions do not have dedicated release notes, their changes are reflected in the regular [DuckDB release notes](https://github.com/duckdb/duckdb/releases)
* core out-of tree extensions tend to live in a repository in `github.com/duckdb/duckdb_⟨ext_name⟩` but the name may vary. See the [full list]({% link docs/extensions/core_extensions.md %}) of core extensions for details.

## Limitations

DuckDB's extension mechanism has the following limitations:
Expand Down

0 comments on commit 20f5f20

Please sign in to comment.