Private/secret databases: database files that are only visible to plugins

[simonw]

In thinking about the best way to implement simonw/datasette-auth-passwords#6 (SQL-backed user accounts for datasette-auth-passwords) I realized that there are a few different use-cases where a plugin might want to store data that isn't visible to regular Datasette users:

• Storing password hashes
• Storing API tokens
• Storing secrets that are used for data import integrations (secrets for talking to the Twitter API for example)

Idea: allow one or more private database files to be attached to Datasette, something like this:

datasette github.db linkedin.db -s secrets.db -m metadata.yml

The secrets.db file would not be visible using any of the Datasette's usual interface or API routes - but plugins would be able to run queries against it.

So datasette-auth-passwords might then be configured like this:

plugins:
  datasette-auth-passwords:
    database: secrets
    sql: "select password_hash from passwords where username = :username"

The plugin could even refuse to operate against a database that hadn't been loaded as a secret database.

[simonw]

The alternative to this would be to use regular databases and control access to them using Authentication and permissions. My concern there is that it's just too easy for someone to mess up their configuration, which would be really bad. I like the idea of a much stronger defense mechanism specifically designed for secrets that should not be exposed.

Outside of secrets, passwords and tokens this mechanism could also be useful for the use-case of using Datasette to power websites - as seen on https://www.niche-museums.com/ and https://www.rockybeaches.com/ - maybe those sites don't want to expose their data through their API but still want to use datasette-template-sql and the graphql() template tag in datasette-graphql to render data.

[simonw]

datasette-graphql currently dispatches requests through the TableView class, so if that couldn't access private databases then it would not be able to either. See also the concept for datasette.get(...) as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe datasette.get(path, allow_private=True).

[simonw]

Naming challenge: secret databases or private databases?

I prefer private. But datasette -p is already taken by --port. datasette -s is currently available.

[simonw]

Forcing people to spell out datasette github.db --private private.db isn't terrible though.

[simonw]

The problem with the term "private" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/

So I think "secret" is a better term for these.

[simonw]

Closing this issue in favour of this one:

• Proposal: Make the _internal database persistent, customizable, and hidden #2157