Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maint: enable SCRAM support #193

Merged
merged 3 commits into from
May 23, 2024
Merged

Conversation

mble-sfdc
Copy link
Contributor

@mble-sfdc mble-sfdc commented May 21, 2024

In order to support SCRAM support for the new Heroku Postgres "Essential" plans, we need to shift from MD5 hashed passwords in auth_file to plain text. This does not materially change the threat model, as anyone with dyno access can read the passwords from the environment just as well as the file.

See: https://www.pgbouncer.org/config.html#authentication-file-format for more.

This commit switches the auth_type to scram-sha-256 and also pushes server_tls_sslmode up to require over prefer.

Why not use a method like auth_query? Exposing something like pg_authid or pg_shadow in a safe way via a SECURITY DEFINER function is extremely challenging in a multi-tenant environment. This may change in the future.

Fixes #155.

Ref: https://gus.my.salesforce.com/a07EE00001rjvVBYAY

Testing Notes:

  • Deploy an application to Heroku that connects to an Essential-tier Postgres instance and inserts random numbers into a test table
  • Add this branch's version of the buildpack with heroku buildpacks:add https://github.com/heroku/heroku-buildpack-pgbouncer.git#17cb11aa1ebabf5d467cb6341c8dfc155533a326
  • Build the app
  • Tail the app logs:
2024-05-21T14:00:58.410834+00:00 app[conn_test.1]: 2024-05-21 14:00:58.410 UTC [16] LOG S-0x565551b1bb80: db1/[email protected]:5432 new connection to server (from 172.16.0.66:44670)
2024-05-21T14:00:58.416864+00:00 app[conn_test.1]: 2024-05-21 14:00:58.416 UTC [16] LOG S-0x565551b1bb80: db1/[email protected]:5432 SSL established: TLSv1.3/TLS_AES_256_GCM_SHA384/ECDH=prime256v1
2024-05-21T14:00:59.385069+00:00 app[conn_test.1]: thread id 4 starting..
2024-05-21T14:00:59.385879+00:00 app[conn_test.1]: 2024-05-21 14:00:59.385 UTC [16] LOG C-0x565551b15800: db1/[email protected]:50376 login attempt: db=db1 user=u4ifhuflluntg3 tls=no
2024-05-21T14:01:02.385909+00:00 app[conn_test.1]: thread id 5 starting..
2024-05-21T14:01:02.386555+00:00 app[conn_test.1]: 2024-05-21 14:01:02.386 UTC [16] LOG C-0x565551b14ae0: db1/[email protected]:52666 closing because: client close request (age=21s)

In order to support SCRAM support for the new Heroku Postgres
"Essential" plans, we need to shift from MD5 hashed passwords in
`auth_file` to plain text. This does not materially change the threat
model, as anyone with dyno access can read the passwords from the
environment just as well as the file.

See: https://www.pgbouncer.org/config.html#authentication-file-format
for more.

This commit switches the `auth_type` to `scram-sha-256` and also pushes
`server_tls_sslmode` up to `require` over `prefer`.

Why not use a method like `auth_query`? Exposing something like
`pg_authid` or `pg_shadow` in a safe way via a `SECURITY DEFINER`
function is extremely challenging in a multi-tenant environment. This
may change in the future.

Fixes #155.

Ref: https://gus.my.salesforce.com/a07EE00001rjvVBYAY
auth_file = $CONFIG_DIR/users.txt
server_tls_sslmode = prefer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect this change to break anything? I suspect not now that we support TLS fleet-wide.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not expect this to break anything - frankly if it does, consumers of the buildpack have bigger problems to worry about. We've been enforcing TLS for some time now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've seen a few GitHub Issues in the past where customers are using the buildpack to connect to RDS/Crunchy/etc. I hope that this change doesn't impact that as well but I would assume they are also supporting TLS and agree customers have bigger problems if they are not for some reason.

Copy link
Contributor

@binarycleric binarycleric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks straight-forward enough to me and I'm glad that we're moving away from MD5ed passwords. I spoke to Troy on Slack and he has verified that the buildpack works against essential-0, mini, and standard-0 addons so we can be confident it'll work against the fleet.

Copy link

@dhagberg-sf dhagberg-sf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the notes on testing - seems quite reasonable.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guessing this will never be used on a multiuser system, but if it's possible, setting a umask up at the top to ensure the users.txt file is ONLY readable by the intended user/group would be a good practice. But this is likely going in a single-user container so prob not necessary.

@dhagberg-sf
Copy link

dhagberg-sf commented May 21, 2024

So double checked that SCRAM has been supported since pg10, so that should not be an issue.

But we'll need to be quite careful in our release notes (significant version bump?) to indicate the database must be configured to support scram auth in pg_hba.conf (which will be true for all heroku dbs, but as indicated above sometimes this buildpack might be used for other db targets).

[edit] wait, this is cool: https://www.postgresql.org/docs/14/auth-password.html

To ease transition from the md5 method to the newer SCRAM method, if md5 is specified as a method in pg_hba.conf but the user's password on the server is encrypted for SCRAM (see below), then SCRAM-based authentication will automatically be chosen instead.

Copy link
Member

@edmorley edmorley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great from a buildpack implementation point of view (I'll leave the Postgres side of the review to others). When this is released, do you think there is a need for a public changelog post on https://devcenter.heroku.com/changelog ?

test/gen-pgbouncer-conf.bats Outdated Show resolved Hide resolved
Changelog.md Show resolved Hide resolved
@troycoll
Copy link

@edmorley we've got a changelog drafted and approved, will publish as soon as this version is released.

@troycoll troycoll merged commit c93b9ee into main May 23, 2024
6 checks passed
@troycoll troycoll deleted the mble-sfdc-janky-scram-sha-256-support branch May 23, 2024 12:17
@edmorley
Copy link
Member

@troycoll Are you ready for me to perform the release?

@troycoll
Copy link

@edmorley yep fire away! I'll publish the Changelog now

edmorley added a commit that referenced this pull request May 23, 2024
edmorley added a commit that referenced this pull request May 23, 2024
@edmorley
Copy link
Member

Released via #194.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SCRAM authentication fails
5 participants