Tech Radar 5

[gratipay-bot]

← Tech Radar 4

Docs

http://inside.gratipay.com/howto/sweep-the-radar

Mission

The mission of the tech team is to support the product team in solving problems in the market.

Scope

• dev
• ops
• PagerDuty
• UptimeRobot
• Heroku
• Digital Ocean
• AWS
• Payday
• Librato
• https://github.com/gratipay/postgres.py
• https://github.com/gratipay/gip.rocks
• https://github.com/gratipay/honcho
• https://github.com/gratipay/environment.py

Roadmap

• stop using immutable ids as primary key (use id as primary key instead of username gratipay.com#835)
• other embarrassments

[chadwhitacre]

https://gratipay.freshdesk.com/helpdesk/tickets/4991

Your database (MAROON on gratipay) must undergo maintenance.

Several vendors recently disclosed some vulnerabilities (CVE-2016-2107/2108) in OpenSSL. You can find more details here. As a result, we need to perform maintenance to your database as soon as possible to apply the patch for this vulnerability. We plan on performing this maintenance at 2016-05-31 19:00:00 +0000 during your set maintenance window of Tuesdays 19:00 to 23:00 UTC.

At that time, we will restart your database or database server as needed.

You may choose a new or more appropriate maintenance window, as needed. For example, you may run: heroku pg:maintenance window="Tuesday 14:30" MAROON to set a maintenance window for Tuesdays at 2:30pm UTC.

You can also run this maintenance directly, using heroku pg:maintenance run MAROON.

This maintenance will stop and restart the Postgres process. When that happens, all of the connections from your app dynos to your database will be closed.

Most app frameworks will gracefully handle database connections going away by retrying until the app can make new database connections. ActiveRecord, for example, should start trying to use connections from the connection pool. It will eventually start throwing errors saying it couldn't get a database connection after 5 seconds. Once the database comes back online, the app should be able to open new connections to the database and continue to serve requests.

However, we've seen some cases where ActiveRecord fills up its connection pool and decides not to try to open any more database connections. In that case, the quickest way to fix it is by running heroku restart, which will restart your dynos.

We recommend that you run the maintenance directly for production applications and make sure that your app is back and running after the maintenance.

If you have any questions regarding this maintenance, please let us know.

[chadwhitacre]

Running maintenance ...

[chadwhitacre]

$ heroku heroku pg:maintenance run
$ heroku pg:maintenance run MAROON
 !    Application must be in maintenance mode or --force flag must be used
$ heroku maintenance
off
$ heroku maintenance on
 ▸    Error: Unexpected argument on
 ▸    Usage: heroku maintenance
 ▸    You gave this command too many arguments. Try the command again without these extra arguments.
 ▸
 ▸    See more information with heroku maintenance --help
$ heroku maintenance --help
Usage: heroku maintenance

display the current maintenance status of app

 -a, --app APP       # app to run command against
 -r, --remote REMOTE # git remote of app to run command against

Additional commands, type "heroku help COMMAND" for more details:

  maintenance:off  #  take the app out of maintenance mode
  maintenance:on   #  put the app into maintenance mode

$ heroku maintenance:on
WARNING: Toolbelt is currently updating
Enabling maintenance mode for ⬢ gratipay... done
$ heroku pg:maintenance run MAROON
Maintenance triggered
$ heroku maintenance:off
Disabling maintenance mode for ⬢ gratipay... done
$ heroku pg:maintenance --help
Usage: heroku pg:maintenance <info|run|window> <DATABASE>

 manage maintenance for <DATABASE>
 info               # show current maintenance information
 run                # start maintenance
   -f, --force      #   run pg:maintenance without entering application maintenance mode
 window="<window>"  # set weekly UTC maintenance window for DATABASE
                     # eg: `heroku pg:maintenance window="Sunday 14:30"`

$ heroku pg:maintenance info
Usage: heroku pg:maintenance <info|run|window> <DATABASE>

 manage maintenance for <DATABASE>
 info               # show current maintenance information
 run                # start maintenance
   -f, --force      #   run pg:maintenance without entering application maintenance mode
 window="<window>"  # set weekly UTC maintenance window for DATABASE
                     # eg: `heroku pg:maintenance window="Sunday 14:30"`

$ heroku pg:maintenance info MAROON
Maintenance in progress, started after 2016-05-25 21:19:08 +0000, window is Tuesdays 19:00 to 23:00 UTC
$

[chadwhitacre]

$ heroku pg:maintenance info MAROON
Maintenance scheduled for 2016-05-25 21:19:08 +0000, required by 2016-06-07 19:39:44 +0000, window is Tuesdays 19:00 to 23:00 UTC
$ 

[chadwhitacre]

https://gratipay.freshdesk.com/helpdesk/tickets/4991

Maintenance on your database (MAROON on gratipay) has been completed and your database is now back online.
We expect operations to continue normally, but feel free to notify us if there are any outstanding issues with your database (MAROON on gratipay).

$ heroku pg:maintenance info MAROON
Maintenance not required, window is Tuesdays 19:00 to 23:00 UTC
$

👍