Skip to content

Commit

Permalink
Link Community Extensions more widely
Browse files Browse the repository at this point in the history
  • Loading branch information
szarnyasg committed Jan 7, 2025
1 parent 54f2eb2 commit 76fd488
Show file tree
Hide file tree
Showing 8 changed files with 15 additions and 80 deletions.
6 changes: 3 additions & 3 deletions community_extensions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ excerpt: |

Welcome to the documentation for the DuckDB Community Extensions.

This website contains all documentation specific to community contributed & maintained extensions for DuckDB.
This website contains all documentation specific to community-contributed extensions for DuckDB.

DuckDB is an analytical in-process SQL database management system, documented at [DuckDB documentation website]({% link docs/index.md %}).

Expand All @@ -30,7 +30,7 @@ The `quack` extension is now loaded and ready to use
SELECT quack('world');
```

## What are Community Extensions
## What Are Community Extensions?

Community Extensions are [DuckDB extensions]({% link docs/extensions/overview.md %}) that are not maintained by the DuckDB team.

Expand All @@ -44,7 +44,7 @@ Check the [Development page]({% link community_extensions/development.md %}) on

DuckDB Community extensions are conceptually similar to a package manager such as [Homebrew](https://brew.sh/), where code will reside in your own repository, but build and distribution is centralized.

## Secure usage of DuckDB Community Extensions
## Security Considerations for Using Community Extensions

DuckDB Community Extensions are signed, so that on LOAD a check is performed to prove a given extensions has been built by the Community Extension CI.

Expand Down
2 changes: 1 addition & 1 deletion docs/api/python/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ con.load_extension("spatial")

### Community Extensions

To load [community extensions]({% link docs/extensions/community_extensions.md %}), use `repository="community"` argument to the `install_extension` method.
To load [community extensions]({% link community_extensions/index.md %}), use `repository="community"` argument to the `install_extension` method.

For example, install and load the `h3` community extension as follows:

Expand Down
68 changes: 2 additions & 66 deletions docs/extensions/community_extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,69 +3,5 @@ layout: docu
title: Community Extensions
---

DuckDB recently launched a [Community Extensions repository](https://github.com/duckdb/community-extensions).
For details, see the [announcement blog post]({% post_url 2024-07-05-community-extensions %}).

## User Experience

We are going to use the [`h3` extension](https://github.com/isaacbrodsky/h3-duckdb) as our example.
This extension implements [hierarchical hexagonal indexing](https://github.com/uber/h3) for geospatial data.

Using the DuckDB Community Extensions repository, you can install and load the `h3` extension as follows:

```sql
INSTALL h3 FROM community;
LOAD h3;
```

Then, you can instantly start using it. Note that the sample data is 500 MB:

```sql
SELECT
h3_latlng_to_cell(pickup_latitude, pickup_longitude, 9) AS cell_id,
h3_cell_to_boundary_wkt(cell_id) AS boundary,
count() AS cnt
FROM read_parquet('https://blobs.duckdb.org/data/yellow_tripdata_2010-01.parquet')
GROUP BY cell_id
HAVING cnt > 10;
```

On load, the extension’s signature is checked, both to ensure platform and versions are compatible, and to verify that the source of the binary is the community extensions repository. Extensions are built, signed and distributed for Linux, macOS, Windows, and WebAssembly. This allows extensions to be available to any DuckDB client using version 1.0.0 and upcoming versions.

For more details, see the [`h3` extension’s documentation]({% link community_extensions/extensions/h3.md %}).

## Developer Experience

From the developer’s perspective, the Community Extensions repository performs the steps required for publishing extensions, including building the extensions for all relevant [platforms]({% link docs/dev/building/overview.md %}#supported-platforms), signing the extension binaries and serving them from the repository.

For the [maintainer of `h3`](https://github.com/isaacbrodsky/), the publication process required performing the following steps:

1. Sending a PR with a metadata file `description.yml` contains the description of the extension:

```yaml
extension:
name: h3
description: Hierarchical hexagonal indexing for geospatial data
version: 1.0.0
language: C++
build: cmake
license: Apache-2.0
maintainers:
- isaacbrodsky

repo:
github: isaacbrodsky/h3-duckdb
ref: 3c8a5358e42ab8d11e0253c70f7cc7d37781b2ef
```
2. The CI will build and test the extension. The checks performed by the CI are aligned with the [`extension-template` repository](https://github.com/duckdb/extension-template), so iterations can be done independently.

3. Wait for approval from the DuckDB Community Extension repository’s maintainers and for the build process to complete.

## Security Considerations

See the [Securing Extensions page]({% link docs/operations_manual/securing_duckdb/securing_extensions.md %}) for details.

## List of Community Extensions

See the [DuckDB Community Extensions repository site](https://duckdb.org/community_extensions).
Community-contributed extensions can be installed from the Community Extensions repository since [summer 2024]({% post_url 2024-07-05-community-extensions %}).
Please visit the [Community Extensions section]({% link community_extensions/index.md }) of the documentation for more details.
5 changes: 2 additions & 3 deletions docs/extensions/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ SELECT extension_name, installed, description
FROM duckdb_extensions();
```


| extension_name | installed | description |
|-------------------|-----------|--------------------------------------------------------------|
| arrow | false | A zero-copy data integration between Apache Arrow and DuckDB |
Expand Down Expand Up @@ -76,7 +75,7 @@ to `LOAD` the spatial extension (again ignoring the statement if it is already l
#### Extension Repository

Optionally a repository can be provided where the extension should be installed from, by appending `FROM <repository>` to the `INSTALL`/`FORCE INSTALL` command.
This repository can either be an alias, such as [`community`]({% link docs/extensions/community_extensions.md %}), or it can be a direct URL, provided as a single-quoted string.
This repository can either be an alias, such as [`community`]({% link community_extensions/index.md %}), or it can be a direct URL, provided as a single-quoted string.

After installing/loading an extension, the [`duckdb_extensions` function](#listing-extensions) can be used to get more information.

Expand All @@ -98,7 +97,7 @@ To see which extensions can be autoloaded, check the [core extensions list]({% l

### Community Extensions

DuckDB supports installing third-party [Community Extensions]({% link docs/extensions/community_extensions.md %}).
DuckDB supports installing third-party [Community Extensions]({% link community_extensions/index.md %}).
These are contributed by community members but they are built, signed, and distributed in a centralized repository.

### Installing Extensions through Client APIs
Expand Down
2 changes: 1 addition & 1 deletion docs/operations_manual/securing_duckdb/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Queries can access the secrets defined in the Secrets Manager. For example, if t

## Locking Configurations

Security-related configuration settings generally lock themselves for safety reasons. For example, while we can disable [community extensions]({% link docs/extensions/community_extensions.md %}) using the `SET allow_community_extensions = false`, we cannot re-enable them again after the fact without restarting the database. Trying to do so will result in an error:
Security-related configuration settings generally lock themselves for safety reasons. For example, while we can disable [Community Extensions]({% link community_extensions/index.md %}) using the `SET allow_community_extensions = false`, we cannot re-enable them again after the fact without restarting the database. Trying to do so will result in an error:

```console
Invalid Input Error: Cannot upgrade allow_community_extensions setting while database is running
Expand Down
6 changes: 3 additions & 3 deletions docs/operations_manual/securing_duckdb/securing_extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ DuckDB offers the following security levels for extensions.

| Usable extensions | Description | Configuration |
|-----|---|---|
| core | Extensions can only be installed from the `core` repository. | `SET allow_community_extensions = false` |
| core and community | Extensions can only be installed from the `core` and `community` repositories. | This is the default security level. |
| `core` | Extensions can only be installed from the `core` repository. | `SET allow_community_extensions = false` |
| `core` and `community` | Extensions can only be installed from the `core` and `community` repositories. | This is the default security level. |
| any extension incl. unsigned | Any extensions can be installed. | `SET allow_unsigned_extensions = true` |

Security-related configuration settings [lock themselves]({% link docs/operations_manual/securing_duckdb/overview.md %}#locking-configurations), i.e., it is only possible to restrict capabilities in the current process.
Expand All @@ -40,7 +40,7 @@ Invalid Input Error: Cannot upgrade allow_community_extensions setting while dat

## Community Extensions

DuckDB has a [Community Extensions repository]({% link docs/extensions/community_extensions.md %}), which allows convenient installation of third-party extensions.
DuckDB has a [Community Extensions repository]({% link community_extensions/index.md %}), which allows convenient installation of third-party extensions.
Community extension repositories like pip or npm are essentially enabling remote code execution by design. This is less dramatic than it sounds. For better or worse, we are quite used to piping random scripts from the web into our shells, and routinely install a staggering amount of transitive dependencies without thinking twice. Some repositories like CRAN enforce a human inspection at some point, but that’s no guarantee for anything either.

We’ve studied several different approaches to community extension repositories and have picked what we think is a sensible approach: we do not attempt to review the submissions, but require that the *source code of extensions is available*. We do take over the complete build, sign and distribution process. Note that this is a step up from pip and npm that allow uploading arbitrary binaries but a step down from reviewing everything manually. We allow users to [report malicious extensions](https://github.com/duckdb/community-extensions/security/advisories/new) and show adoption statistics like GitHub stars and download count. Because we manage the repository, we can remove problematic extensions from distribution quickly.
Expand Down
2 changes: 1 addition & 1 deletion docs/sql/statements/load_and_install.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Install the [`httpfs`]({% link docs/extensions/httpfs/overview.md %}) extension:
INSTALL httpfs;
```

Install the `h3` [Community Extension]({% link docs/extensions/community_extensions.md %}):
Install the [`h3` Community Extension]({% link community_extensions/extensions/h3.md %}):

```sql
INSTALL h3 FROM community;
Expand Down
4 changes: 2 additions & 2 deletions faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,11 @@ Please consult the [trademark guidelines for DuckDB™]({% link trademark_guidel

<div class="answer" markdown="1">

Features in DuckDB can be implemented in different ways: in the main DuckDB project, as a [core extension]({% link docs/extensions/core_extensions.md %}) or a [community extension]({% link docs/extensions/community_extensions.md %}). We recommend following these guidelines for feature requests:
Features in DuckDB can be implemented in different ways: in the main DuckDB project, as a [core extension]({% link docs/extensions/core_extensions.md %}) or a [community extension]({% link community_extensions/index.md %}). We recommend following these guidelines for feature requests:

* If you would like a feature to be implemented in DuckDB, please raise and issue in the [Ideas section in DuckDB's GitHub Discussions forum](https://github.com/duckdb/duckdb/discussions/categories/ideas). The DuckdB team monitors these ideas and, over time, implements the frequently requested features. For example, we recently published the [Avro Community Extension]({% link community_extensions/extensions/avro.md %}) to support reading Avro files, which was the most requested feature in the issue tracker.
* If you would like to implement a feature in the main DuckDB project, please discuss it with the DuckDB team on GitHub Discussions or on [our Discord server](https://discord.duckdb.org/). The team can verify whether the idea and the proposed implementation line up with the project's long-term vision.
* If you would like to implement a feature as an extension, consider submitting it to the [Community Extensions repository]({% link docs/extensions/community_extensions.md %}).
* If you would like to implement a feature as an extension, consider submitting it to the [Community Extensions repository]({% link community_extensions/index.md %}).

Please note that DuckDB Labs, the company that employs the main DuckDB contributors, provides [consultancy services for DuckDB](https://duckdblabs.com/support/), which can include implementing features in DuckDB or as DuckDB extensions.

Expand Down

0 comments on commit 76fd488

Please sign in to comment.