Merge pull request #2844 from lsst-sqre/tickets/DM-42337

DM-42337: Improve Nublado documentation

docs/applications/nublado/troubleshoot.rst

@@ -17,29 +17,13 @@ In the resulting JSON document, ``config`` shows the current operative configura
 Clear session database entry
 ============================
 
-Sometimes JupyterHub and its session database will get into an inconsistent state where it thinks a pod is already running but cannot shut it down.
-The typical symptom of this is that spawns for that user fail with an error saying that the user's lab is already pending spawn or pending deletion, but the user cannot connect to their pod.
+Historically, we sometimes saw JupyterHub get into an inconsistent state where it thought a pod was already running and couldn't be shut down.
+We haven't seen this problem since switching to the Nublado controller, but it may still be possible for the JupyterHub session database to get out of sync.
 
-Recovery may require manually clearing the user's entry in the session database as follows:
+If JupyterHub keeps telling a user that their lab is already spawning or shutting down, but doesn't allow them to connect to the lab or shut it down, following the instructions on `deleting a user session <https://nublado.lsst.io/admin/delete-user-session.html>`__ may fix the problem.
 
-#. Remove the user's lab namespace, if it exists.
-
-#. Remove the user from the session database.
-   First, connect to the database:
-
-   .. code-block:: shell
-
-      pod=$(kubectl get pods -n postgres | grep postgres | awk '{print $1}')
-      kubectl exec -it -n postgres ${pod} -- psql -U jovyan jupyterhub
-
-   Then, at the PostgreSQL prompt:
-
-   .. code-block:: sql
-
-      delete from users where name='<user-to-remove>'
-
-In some cases you may also need to remove the user from the spawner table.
-To do this, run ``select * from spawners`` and find the pod with the user's name in it, and then delete that row.
+If it does, investigate how JupyterHub was able to get stuck.
+This indicates some sort of bug in Nublado.
 
 Prepuller is running continuously and/or expected menu items are missing
 ========================================================================

docs/applications/nublado/upgrade.rst

@@ -8,8 +8,9 @@ Most of the time, upgrading Nublado can be done simply by syncing the applicatio
 There will be a brief outage for spawning new pods, but users with existing pods should be able to continue working.
 
 Occasionally, new versions of JupyterHub will require a schema update.
-We do not routinely enable automatic schema updates currently, so JupyterHub will refuse to start if a database schema update is required.
-To enable schema updates, add:
+Automatic schema updates are off by default, so JupyterHub will refuse to start if a database schema update is required.
+
+To enable schema updates, add the following to :file:`values-{environment}.yaml` for the ``nublado`` application:
 
 .. code-block:: yaml
 
@@ -18,7 +19,10 @@ To enable schema updates, add:
        db:
          upgrade: true
 
-(The ``jupyterhub`` and ``hub`` keys probably already exist in the ``values-<environment>.yaml`` file, so just add the ``db.upgrade`` setting in the correct spot.)
-Then, JupyterHub will automatically upgrade its database when the new version starts.
-You can then remove this configuration again if you're worried about automatic updates misbehaving later.
-Alternatively, if there's a schema update, it's probably a pretty major upgrade to JupyterHub, and it may be a better idea to shut down the Hub, remove all user namespaces, and then connect to the database and drop all tables; when the Hub is restarted, the correct schema will be generated.  Obviously this will boot all users from the running system, but that may be appropriate for major upgrades.
+(The ``jupyterhub`` and ``hub`` keys probably already exist, so just add the ``db.upgrade`` setting in the correct spot.)
+JupyterHub will then automatically upgrade its database when it is restarted running the new version.
+
+You can then this configuration afterwards if you're worried about applying a schema update without being aware that you're doing so.
+
+Alternately, for major upgrades to JupyterHub, you can choose to start from an empty database.
+To do this, follow the instructions in the `Nublado documentation on wiping the database <https://nublado.lsst.io/admin/wipe-database.html>`__.