DM-46556 Add consdb documentation #51

Vebop · 2024-11-08T02:43:34Z

Added Deployment process documentation to Schema Migration Process in the operator guide.
Added other small documentation, like schema versioning conversation, and monitoring notes.
Most recent build of docs: https://consdb.lsst.io/builds/105

JeremyMcCormick · 2024-11-11T10:35:27Z

doc/operator-guide/schema-migration-process.rst

+
+    pip install lsst-felis
+
+    felis validate [options] [edited cdb_*.yml files]


Technically, this does not check that SQL tables can be created. Rather, the validate command checks that the schema conforms to Felis's data model. Currently, schemas could pass validation but database creation may still fail. (Additional validation functions are planned to be added in order to rectify this.)

If you want to check that a database can be created, the felis create command should be used.

Note that the sdm_schemas checks on PRs will create test databases for all of the schemas in MySQL, PostgreSQL, and SQLite, so it may not be necessary to check this locally. Database creation failures will be caught in CI. (Running the validate command locally is still a good idea though the CI does this as well.)

By the way, shell globs work fine with Felis commands, so this would work to validate all cdb schemas:

cd sdm_schemas && felis validate yml/cdb_*.yaml

It makes sense to leave those details to the CI and Felis documentation then. I'm updating accordingly.

JeremyMcCormick · 2024-11-12T10:15:18Z

alembic-autogenerate.py

@@ -3,27 +3,28 @@
 #
 # How to use this script:
 # 1. Load the LSST environment and setup sdm_schemas and felis.
-#        source loadLSST.bash
+#        source loadLSST.bash or activate your eups setup


Alternately, the following should work to install all necessary dependencies for generating the migrations:

pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml

And sdm_schemas could then be setup with:

git clone https://github.com/lsst/sdm-schemas cd sdm_schemas export SDM_SCHEMAS_DIR=`pwd`

I just verified that the migration script runs okay with the above.

It shouldn't really be necessary to have a full LSST stack available just to run the migrations. (Neitheralembic-automigratte.py nor the env.py script depend on any stack software besides felis and sdm_schemas.)

Should be able to pip install sdm_schemas itself and use the python package resources to find the schemas.

Should be able to pip install sdm_schemas itself and use the python package resources to find the schemas.

That is on the todo list.

Thank you for verifying your directions above, very helpful!

alembic-autogenerate.py

doc/operator-guide/schema-migration-process.rst

alembic-autogenerate.py

JeremyMcCormick · 2024-11-12T10:41:38Z

doc/user-guide/schemas.rst

+
+    Think about the utility of these versions in terms of interaction with the ConsDB APIs, migrations, etc.
+
+    Do sdm_schemas versions appear in the db?


The versions are not currently stored in the database, but I plan to add this capability.

See DM-47367

JeremyMcCormick · 2024-11-12T10:43:09Z

doc/user-guide/schemas.rst

+    Currently the schemas are tagged and versioned as a set, at least w.r.t. the Science Platform.
+    So once ConsDB is available in TAP, it should be part of that set.
+
+    What do users see, how does TAP play into this, do the schema need this type of micro versioning?


We do not currently have any capability to provide individual schema versions via TAP, as TAP_SCHEMA does not include a column providing this information. Users on the Science Platform do not really see any schema versions at all, currently, because the tag on sdm_schemas is only specified in Phalanx and not propagated to the web UI either.

This section was added just to capture the conversation that was happening in reference to schema versioning the other week. Is there still clarification I need to add here, or maybe we need to speak about this at a weekly meeting? Did this happen on 11.11, when I missed?

Unless the procedures change, ConsDB schemas will be deployed to the Science Platform as part of an sdm_schemas release. This version will not be visible to users. This will determine what tables and columns are visible in TAP. We can cut new releases of sdm_schemas to incorporate changes to ConsDB as needed. (The sdm_schemas semver releases used on the Science Platform are made ad hoc/as needed and not on a fixed schedule.)

We will have the capability to deploy different releases of sdm_schemas to different environments if this is required (usdf-rsp, usdf-rsp-dev, etc.). But we cannot currently deploy different versions of specific schemas individually. They are versioned as a set.

JeremyMcCormick · 2024-11-12T10:47:00Z

doc/user-guide/schemas.rst

+
+    - or there can be ones that are not neutral.
+
+    Think about the utility of these versions in terms of interaction with the ConsDB APIs, migrations, etc.


I would think that a release of the ConsDB API should be connected to a set of compatible schema versions. Would a tag of sdm_schemas be good enough? We don't currently have a good way of retrieving schemas by their internal versions, so I don't know how useful it would be to have them listed individually. This sdm_schemas tag could be included as part of the release notes.

I am not sure how this information could be tracked automatically though. Maybe it is just something that needs to be tracked and included manually.

bbrondel · 2024-11-12T15:00:52Z

doc/operator-guide/deployment.rst

+    Deployments of the Consolidated Database are currently located at
+
+      -  Summit
+      -  USDF (+ dev)


It's worth noting here that (1) USDF and USDF-dev use the same underlying database and (2) that database is a replica of the summit database.

I will add a clarification for this.

JeremyMcCormick · 2024-11-12T15:37:44Z

doc/.DS_Store

This file should be removed and probably added to .gitignore.

JeremyMcCormick · 2024-11-12T15:41:07Z

As a general comment, the RST formatting needs work.

See cheat sheet here.

doc/operator-guide/deployment.rst

doc/operator-guide/monitoring.rst

doc/operator-guide/schema-migration-process.rst

JeremyMcCormick

I left some specific suggestions for changes.

Overall, I'm wondering why all the additional content is being added that doesn't have to do with the deployment process. I would think going forward that it would be better to stick more closely to the issue a ticket is intended to solve in updating the documentation, rather than including a bunch of different updates that are not closely related. (I'm not requesting that these changes be removed this time, though.)

JeremyMcCormick · 2024-12-04T20:00:14Z

alembic-autogenerate.py

-#        setup felis
-#        setup -r /path/to/sdm_schemas
+# 1. Install Felis and sdm_schemas, set environment variables:
+#        pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml


This seems to be missing a few dependencies. What I found worked from scratch was:

pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml black psycopg2-binary

JeremyMcCormick · 2024-12-04T20:01:08Z

alembic-autogenerate.py

-#        python alembic-autogenerate.py this is my revision message "\n" \
-#            the message can span multiple lines "\n" \
-#            if desired
+#        python alembic-autogenerate.py DM-12345


I don't know that we have "officially" determined that ticket names will be used in migrations. Maybe it needs to be discussed in the meeting first?

Did we officially decide something? it wasn't super clear to me from the recording

doc/operator-guide/deployment.rst

JeremyMcCormick · 2024-12-04T20:02:20Z

doc/operator-guide/deployment.rst

+
+Process:
+--------
+


Shouldn't this section have content if the PR is for adding deployment process documentation?

doc/operator-guide/monitoring.rst

doc/operator-guide/runbook.rst

doc/operator-guide/schema-migration-process.rst

JeremyMcCormick · 2024-12-04T20:05:10Z

doc/operator-guide/schema-migration-process.rst

@@ -2,8 +2,118 @@
 Schema Migration Process


Bullets also overused in this section. I would remove most of them and just write it as prose.

JeremyMcCormick · 2024-12-04T20:05:42Z

doc/user-guide/schemas.rst

-        * Near-real-time "prompt" ConsDB replicates a subset of the USDF version
-        * Data Release ConsDB is a snapshot of a subset of the USDF version with data pertaining to the exposures/visits in the DR
-* Schema browser
+Types of schemas


Rewrite the bullets below as complete sentences.

JeremyMcCormick · 2024-12-04T20:21:53Z

@Vebop Additional note: If you want to make various changes/updates across the whole documentation site, I do not feel that this needs to be connected to a ticket. I think it would be reasonable to do this on user branches. Tickets can be used for more focused/specific work.

Unless you want to create and use a generic ticket like "Write consdb documentation site." :)

JeremyMcCormick · 2024-12-05T00:59:55Z

doc/operator-guide/runbook.rst

+
+#consolidated-database
+
+Overall complaints:


This seems somewhat terse and could use more explanation. Is the idea that someone would open a ticket and assign it to a certain individual based on the specific sub-component?

I copied this from the linked runbook, so I'm actually not sure.
https://rubinobs.atlassian.net/wiki/spaces/LSSTOps/pages/45665320/Consolidated+Database+ConsDB+Runbook+draft+incomplete

JeremyMcCormick · 2024-12-05T01:00:34Z

doc/operator-guide/schema-migration-process.rst

+Create an Alembic Migration (manually)
+======================================
+
+- Alembic (https://alembic.sqlalchemy.org/en/latest/front.html) keeps track of versioning by autogenerated migrations to sync the test stands and summit databases.


Make "Alembic" a proper link to the website.

JeremyMcCormick · 2024-12-05T01:01:16Z

doc/operator-guide/schema-migration-process.rst

+- Access to argo-cd deployments is available via the Summit OpenVPN.
+- To coordinate your deployment update on the summit, you must attend Coordination Activities Planning (CAP) meeting on Tuesday mornings and announce your request.
+
+- Add it to the agenda here: https://rubinobs.atlassian.net/wiki/spaces/LSSTCOM/pages/53765933/Agenda+Items+for+Future+CAP+Meetings


JeremyMcCormick · 2024-12-05T01:01:26Z

doc/operator-guide/schema-migration-process.rst

+- The CAP members may tell you a time frame that is acceptable for you to perform these changes.
+
+- They may also tell you specific people to coordinate with to help you take images to test LATISS and LSSTCOMCAMSIM tables. There will be more tables to test eventually.
+- Channels to note: #rubinobs-test-planning; #summit-announce; #summit-auxtel, https://obs-ops.lsst.io/Communications/slack-channel-usage.html.


JeremyMcCormick · 2024-12-05T01:05:55Z

@Vebop Another general comment...

There is a mix of URLs and linking styles in this PR, some of which do not work.

Links should be consistently formatted like this in RST:

`International Virtual Observatory Alliance <https://ivoa.net/>`__

We should avoid raw URLs and use the above format instead.

alembic-autogenerate.py

…notes

Vebop force-pushed the tickets/DM-46556-deployment branch from e9117f0 to 4feb271 Compare November 8, 2024 17:36

Vebop requested review from ktlim, bbrondel and JeremyMcCormick November 8, 2024 19:43

JeremyMcCormick changed the title ~~Tickets/dm 46556 deployment~~ DM-46556 Add deployment process documentation Nov 11, 2024

JeremyMcCormick reviewed Nov 11, 2024

View reviewed changes

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

alembic-autogenerate.py Outdated Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

doc/operator-guide/schema-migration-process.rst Outdated Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

alembic-autogenerate.py Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

bbrondel reviewed Nov 12, 2024

View reviewed changes

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

doc/operator-guide/deployment.rst Outdated Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

doc/operator-guide/monitoring.rst Outdated Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

doc/operator-guide/schema-migration-process.rst Outdated Show resolved Hide resolved

JeremyMcCormick reviewed Nov 12, 2024

View reviewed changes

doc/operator-guide/schema-migration-process.rst Outdated Show resolved Hide resolved

Vebop force-pushed the tickets/DM-46556-deployment branch 2 times, most recently from da8576b to f4b66f1 Compare November 25, 2024 03:15

Vebop marked this pull request as ready for review November 25, 2024 03:22

Vebop requested review from JeremyMcCormick and bbrondel November 25, 2024 23:49

JeremyMcCormick requested changes Dec 4, 2024

View reviewed changes

JeremyMcCormick reviewed Dec 5, 2024

View reviewed changes

Vebop commented Dec 12, 2024

View reviewed changes

alembic-autogenerate.py Outdated Show resolved Hide resolved

Vebop changed the title ~~DM-46556 Add deployment process documentation~~ DM-46556 Add consdb documentation Jan 3, 2025

Vebop added 4 commits January 3, 2025 14:05

Add Schema migration process documentation and update alembic script …

33b3120

…notes

Add startracker tables and migrations for quicklook updates

90282c5

Add Schema migration process documentation and update alembic script …

fe52e9f

…notes

Update links formatting and other PR comments

d6c9f8f

Vebop force-pushed the tickets/DM-46556-deployment branch from 828d909 to d6c9f8f Compare January 3, 2025 21:05

Updated wording in schemas.rst

828aac0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DM-46556 Add consdb documentation #51

DM-46556 Add consdb documentation #51

Vebop commented Nov 8, 2024 •

edited

Loading

JeremyMcCormick Nov 11, 2024 •

edited

Loading

JeremyMcCormick Nov 16, 2024

Vebop Nov 18, 2024

JeremyMcCormick Nov 12, 2024 •

edited

Loading

timj Nov 12, 2024

JeremyMcCormick Nov 12, 2024 •

edited

Loading

Vebop Nov 18, 2024

JeremyMcCormick Nov 12, 2024

JeremyMcCormick Nov 12, 2024 •

edited

Loading

Vebop Nov 18, 2024

JeremyMcCormick Dec 4, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

bbrondel Nov 12, 2024

Vebop Nov 18, 2024

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick commented Nov 12, 2024 •

edited

Loading

JeremyMcCormick left a comment •

edited

Loading

JeremyMcCormick Dec 4, 2024

JeremyMcCormick Dec 4, 2024

Vebop Dec 13, 2024

JeremyMcCormick Dec 4, 2024

JeremyMcCormick Dec 4, 2024

JeremyMcCormick Dec 4, 2024 •

edited

Loading

JeremyMcCormick commented Dec 4, 2024

JeremyMcCormick Dec 5, 2024 •

edited

Loading

Vebop Jan 3, 2025

JeremyMcCormick Dec 5, 2024 •

edited

Loading

JeremyMcCormick Dec 5, 2024

JeremyMcCormick Dec 5, 2024

JeremyMcCormick commented Dec 5, 2024 •

edited

Loading


		pip install lsst-felis

		felis validate [options] [edited cdb_*.yml files]


		Think about the utility of these versions in terms of interaction with the ConsDB APIs, migrations, etc.

		Do sdm_schemas versions appear in the db?


		- or there can be ones that are not neutral.

		Think about the utility of these versions in terms of interaction with the ConsDB APIs, migrations, etc.


		Process:
		--------


		#consolidated-database

		Overall complaints:

DM-46556 Add consdb documentation #51

Are you sure you want to change the base?

DM-46556 Add consdb documentation #51

Conversation

Vebop commented Nov 8, 2024 • edited Loading

JeremyMcCormick Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

JeremyMcCormick Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

JeremyMcCormick commented Nov 12, 2024 • edited Loading

JeremyMcCormick left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

JeremyMcCormick commented Dec 4, 2024

JeremyMcCormick Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeremyMcCormick commented Dec 5, 2024 • edited Loading

Vebop commented Nov 8, 2024 •

edited

Loading

JeremyMcCormick Nov 11, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick Dec 4, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick Nov 12, 2024 •

edited

Loading

JeremyMcCormick commented Nov 12, 2024 •

edited

Loading

JeremyMcCormick left a comment •

edited

Loading

JeremyMcCormick Dec 4, 2024 •

edited

Loading

JeremyMcCormick Dec 5, 2024 •

edited

Loading

JeremyMcCormick Dec 5, 2024 •

edited

Loading

JeremyMcCormick commented Dec 5, 2024 •

edited

Loading