Initial push #45
Initial push #45
Conversation
There was a problem hiding this comment.
I made a few inline comments.. I do have a general comment. I think the title is broader than the overall context we intend this to be, or maybe I misunderstood the context.
My impression is that this was supposed to handle the user environment + script queue access to a set of tssw stack packages (ts_observatory_control, ts_standardscripts, ts_externalscripts, etc..). But the title, introduction and some of the content seems to suggest a much broader context (e.g. CSC versioning). I think we need to align these expectations.
As you will see in my comments below, I propose a title change. Also I think we could call this "Observing Environment" which consists of the environment used by observers in nublado and the ScriptQueue. This allows us to differentiate CSCs from OE. I made a suggestion for the first paragraph of the introduction based on this proposal.
Observing-Interface-Setup/production environment_package_management.rst
Outdated
Show resolved
Hide resolved
Observing-Interface-Setup/production environment_package_management.rst
Outdated
Show resolved
Hide resolved
Observing-Interface-Setup/production environment_package_management.rst
Outdated
Show resolved
Hide resolved
|
|
||
| This command will move your current ``~/notebooks/.user_setups`` file to your home directory with a timestamp, then create a new file which points to the base environment. | ||
|
|
||
| Use of the deploy branch |
|
|
||
| The base environment is defined by a list of tags, or commit hashes, representing the packages which are deemed to be stable. | ||
| Note that this list needs to be maintained daily, and can only be updated by the production environment maintainers. | ||
| The list itself is stored in ``/opt/obs_user/base_environment.yaml`` (FIXME-better way to do this? CSV file with package name and branch (or commit hash?)). |
There was a problem hiding this comment.
This information is already in the cycle build, I think instead of having two sources of truth, we should probably use the same file as in the cycle build. I think we could either clone the repository here or copy the one file with the information we want, that is cycle/cycle.env.
There was a problem hiding this comment.
I think I addressed this above?
| The default for each package should be a tagged commit that has been merged to the deploy branch. | ||
| However, in certain cases it may be a specific commit of the deploy branch, specifically if bug-fixes have been applied that are not yet incorporated into the main branch. | ||
|
|
||
| Question: How do we update this when a new cycle build occurs? Just part of a procedure? FIXME |
There was a problem hiding this comment.
We could have a service in Kubernetes that does this.. Basically an app which updates the source file anytime cycle build is updated..
There was a problem hiding this comment.
Hmm, maybe? I'd be a little worried that a "bot" can update the file that might have had something special in it written by a user? One can imagine that if you made a new revision and broke something, then you make that new broken version your base environment
Need to chew on that I guess.
| Question: How do we update this when a new cycle build occurs? Just part of a procedure? FIXME | ||
|
|
||
|
|
||
| On-sky testing then rolling back a CSC |
There was a problem hiding this comment.
I am confused, I thought this was only for the user environment and the ScriptQueue, not for CSCs...
There was a problem hiding this comment.
True, but I was thinking of a use-case, which may just be completely separate, where we deploy a new CSC for people to try, but then give instructions on how they can roll back if they hit a bug, without a 2am phone call.
It's sort of related to "resetting back to the base environment" -- but more for CSCs.
| .. _Production-Environment-Package-Management: | ||
|
|
||
| ######################################### | ||
| Production Environment Package Management |
There was a problem hiding this comment.
How about?
Observing Environment Package Management
There was a problem hiding this comment.
I didn't like my title either :D
So long it implies both the summit, TTS, and BTS then I think it's fine... and I think your suggestion does that.
|
|
||
| .. This section should provide a brief, top-level description of the procedure's purpose and utilization. Consider including the expected user and when the procedure will be performed. | ||
|
|
||
| This page explains how package management in controlled on production environments, specifically the summit. |
There was a problem hiding this comment.
Looking at this from an "Observing Environment Package Management" point of view, consider the following update to this paragraph.
This page explains how package management is controlled on the Observing Environment, specifically the summit.
The Observing Environment is mainly composed of the ScriptQueue and users nublado instance, which are responsible for driving the observatory operations.
(new paragraph?)
Even though the Observing Environment is initially built and deployed alongside the rest of the Observatory Control System components (e.g. CSCs), it needs to follow a slightly different management approach to support on-the-fly updates.
Nominally, the CSC versions and supporting software packages deployed on the summit are managed by thecycle build <https://ts-cycle-build.lsst.io/>_.
Nevertheless, the procedure to deploy hot-fixes for CSCs (e.g. create an alpha tag on the package, update the cycle build and redeploy), is not suitable for the Observing Environment, which requires a much larges set of packages and therefore, longer build times.
These patches to the Observing Environment needs to be rapidly rolled out to the summit and must be simultaneously available to the ScriptQueue CSC as well as the observer's notebook environments.
Most importantly, when testing new software on the summit, which may not be entirely stable, it is critical to have a mechanism to immediately roll back all packages to a designated stable version.
We call this the base observing environment.
There was a problem hiding this comment.
Used most of this... but changed a few things.
| These fixes must be simultaneously available to the ScriptQueue CSC as well as the observer's notebook environments. | ||
| Most importantly, when testing new software on the summit, which may not be entirely stable, it is critical to have a mechanism to immediately roll back all packages to a designated stable version. | ||
| We call this the **base environment**. | ||
| Note that this version may include bug fixes and/or new functionalities that are not included in the previously released cycle build. |
There was a problem hiding this comment.
I think, ideally, the "base environment" would match a revision of the cycle build. This means we have a unified way to control what we deploy at the summit. So the procedure to update the "base environment" would basically be to create a new revision. Now, this doesn't mean we have to build a new version of the SQ and nublado to get it deployed.
There was a problem hiding this comment.
"basically create a new revision" --> What you really mean is update a local version of cycle.env, then have some sort of process that matches the NFS mounted packages to that file, yes?
I think for clarity we really should refer to cycle.env as something else though...
Also, we should strip out all the packages (e.g. CSCs) that would be no-ops as they'll just add confusion.
patrickingraham
force-pushed
the
tickets/DM-36686
branch
2 times, most recently
from
027a6d2 to
17b083c
Compare
patrickingraham
force-pushed
the
tickets/DM-36686
branch
from
17b083c to
21262ba
Compare
No description provided.