OPSIM-1107 Documentation restructure

.github/workflows/build_docs.yaml

@@ -30,7 +30,7 @@ jobs:
         run: |
           mamba install --quiet --file=requirements.txt          
           mamba install --quiet pip
-          pip install "documenteer[pipelines]"
+          pip install "documenteer[guide]"
 
       - name: install rubin_sim
         shell: bash -l {0}
@@ -66,16 +66,14 @@ jobs:
         shell: bash -l {0}
         run: |
           export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
-          cd doc
-          python metric_list.py
+          cd docs
           make html
-          #package-docs build
 
       - name: upload documentation
         uses: lsst-sqre/ltd-upload@v1
         with:
           project: "rubin-sim"
-          dir: "doc/_build/html"
+          dir: "docs/_build/html"
           username: ${{ secrets.ltd_username }}
           password: ${{ secrets.ltd_password }}
 

.github/workflows/ruff.yaml

@@ -13,12 +13,12 @@ jobs:
   isort:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: isort/isort-action@v1
         with:
           requirements-files: "requirements.txt test-requirements.txt"
   ruff:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: chartboost/ruff-action@v1

.github/workflows/test_and_build.yaml

@@ -45,7 +45,7 @@ jobs:
         run: |
           echo `pwd`
           ls ${{ github.workspace }}
-          python -m pip install . --no-deps
+          python -m pip install -e . --no-deps
 
       - name: Access rubin-sim-data cache
         id: cache-rs
@@ -70,11 +70,14 @@ jobs:
         shell: bash -l {0}
         run: conda list
 
+      - name: run black
+        shell: bash -l {0}
+        run: black --check
+
       - name: run tests
         shell: bash -l {0}
         run: |
           export RUBIN_SIM_DATA_DIR=~/rubin_sim_data
-          #pytest -r a -v 
           pytest -r a -v --cov=rubin_sim --cov=tests --cov-report=xml --cov-report=term --cov-branch
 
       - name: Upload coverage to codecov

README.md

@@ -12,102 +12,33 @@ Scheduler, survey strategy analysis, and other simulation tools for Rubin Observ
 [![DOI](https://zenodo.org/badge/365031715.svg)](https://zenodo.org/badge/latestdoi/365031715)
 
 
-## Installation
-
-### Conda Installation ###
-
-If you are only running `rubin_sim` code and not making changes. If you will be editing the code or need the very latest verison, use the pip instructions below.
-```
-conda create -n rubin-sim -c conda-forge rubin_sim  ## Create a new environment and install rubin_sim
-conda activate rubin-sim
-rs_download_data  ## Downloads a few of data to $RUBIN_SIM_DATA_DIR (~/rubin_sim_data if unset)
-conda install -c conda-forge jupyter  ## Optional install of jupyter
-```
-Note that this is not the best option for developers working on their own metrics - a pip installation from their own fork of the repo may work better.
-
-### Pip installation ###
-
-```
-pip install rubin-sim
-```
-
-Please note that the pip installation of pyoorb does not come with the necessary data files. 
-To actually use pyoorb, the data files are most easily installable via conda with
- ```
- conda install -c conda-forge openorb-data
- conda install -c conda-forge openorb-data-de405
- ```
-The pip installation of `rubin_sim` will install the pip version of `pyoorb` which is
-more up-to-date compared to the conda-forge version of `openorb`. For the purposes of 
-`rubin_sim`, the functionality is essentially the same however.
-
-
-### Developer Installation ###
-
-To install `rubin_sim` from source using pip, with all dependencies (including jupyter):
-```
-git clone https://github.com/lsst/rubin_sim.git ; cd rubin_sim  ## clone and cd into repo
-conda create -n rubin-sim --file=all_req.txt   ## optional (but recommended) new conda env
-conda activate rubin-sim   ## substitute mamba for conda if you like
-pip install -e . --no-deps
-rs_download_data  ## Downloads a few GB of data to $RUBIN_SIM_DATA_DIR (~/rubin_sim_data if unset)
-```
-Note that external collaborators will likely want to follow similar directions, using a fork of our rubin_sim github repo first (and then clone from there).
-
-### Data download for rubin_sim ###
-
-**Optional: Set $RUBIN_SIM_DATA_DIR data directory.** By default, `rubin_sim` will download needed data files to `$HOME/rubin_sim_data`. If you would like the data to save elsewhere, you should set the `RUBIN_SIM_DATA_DIR` environment variable. In bash  `export RUBIN_SIM_DATA_DIR="/my/preferred/data/path"` (note, always make sure this is set before trying to run `rubin_sim` packages, so put in your .bashrc or whatnot). Another possibility is to set the location via sym-link, `ln -s /my/preferred/data/path ~/rubin_sim_data`.
-
-```
-export RUBIN_SIM_DATA_DIR=$HOME/rubin_sim_data  ## Optional. Set the data directory path via env variable
-rs_download_data  ## Downloads a few GB of data to $RUBIN_SIM_DATA_DIR
-```
-If you are only interested in a subset of the data, you can specify which directories to download, e.g.
-```
-rs_download_data  --dirs "throughputs,skybrightness,tests,maps"
-```
-
-If you have a previous installation of rubin_sim or wish to update your data download, the flag `--force` will force an update of the data in the relevant $RUBIN_SIM_DATA_DIR directories. 
-
-
-**Example notebooks** to test and further explore rubin_sim, are available at [rubin_sim_notebooks](https://github.com/lsst/rubin_sim_notebooks). 
-```
-git clone https://github.com/lsst/rubin_sim_notebooks.git
-cd rubin_sim_notebooks
-# Example: make a plot of the number of visits per pointing
-jupyter notebook maf/tutorial/Survey_footprint.ipynb  
-```
-
-
-### Downloading additional skybrightness_pre skybrightness files ###
-
-The default skybrightness_pre directory downloaded above contains only one month of pre-calculated skybrightness files.
-If you wish to run the scheduler for a longer time period, or need this information outside of the span of that month period,
-you will need to download a larger set of pre-computed sky data.
-
-To download the entire optional set all the (43 Gb) of pre-computed sky data. 
-```
-rs_download_sky
-```
-Note that subsets of this data can get downloaded via http directly from
-```
-https://s3df.slac.stanford.edu/data/rubin/sim-data/sims_skybrightness_pre/h5_2023_09_12/
-```
-(the file names reflect the range of MJD covered within each data file).
-
-
-## Documentation
-
-Online documentation is available at https://rubin-sim.lsst.io
-Example jupyter notebooks can be found at:  https://github.com/lsst/rubin_sim_notebooks
-
-To create a local build of the documentation:
-```
-conda install -c conda-forge lsst-documenteer-pipelines
-cd doc
-make html
-```
-
-## Getting Help ##
-
-Questions about `rubin_sim` can be posted on the [sims slack channel](https://lsstc.slack.com/archives/C2LQ5JW9W), or on https://community.lsst.org/ (tag @yoachim and/or @ljones so we get notifications about it).
+## rubin_sim ## 
+
+The [Legacy Survey of Space and Time](http://www.lsst.org) (LSST)
+is anticipated to encompass around 2 million observations spanning a decade,
+averaging 800 visits per night. The `rubin_sim` package was built to help
+understand the predicted performance of the LSST.
+
+The `rubin_sim` package contains the following main modules: 
+* `phot_utils` - provides synthetic photometry
+using provided throughput curves based on current predicted performance.
+* `skybrightness` incorporates the ESO
+sky model, modified to match measured sky conditions at the LSST site,
+including an addition of a model for twilight skybrightness. This is used
+to generate the pre-calculated skybrightness data used in
+[`rubin_scheduler.skybrightness_pre`](https://rubin-scheduler.lsst.io/skybrightness-pre.html).
+* `moving_objects` provides a way to generate
+synthetic observations of moving objects, based on how they would appear in
+pointing databases ("opsims") created by
+[`rubin_scheduler`](https://rubin-scheduler.lsst.io).
+* `maf` the Metrics Analysis Framework, enabling efficient and
+scientifically varied evaluation of the LSST survey strategy and progress
+by providing a framework to enable these metrics to run in a
+standardized way on opsim outputs.
+
+More documentation for `rubin_sim` is available at 
+[https://rubin-sim.lsst.io](https://rubin-sim.lsst.io), including installation instructions. 
+
+### Getting Help ###
+
+Questions about `rubin_sim` can be posted on the [sims slack channel](https://lsstc.slack.com/archives/C2LQ5JW9W), or on https://community.lsst.org/c/sci/survey_strategy/ (optionally, tag @yoachim and/or @ljones so we get notifications about it).

README_dev.md

@@ -53,10 +53,12 @@ To update the source contents of the data files:
 
 Process for updating pre-computed files if system throughputs change.
 
-1) update rubin_sim_data/throughputs files
-2) update rubin_sim/rubin_sim/utils/sys_eng_vals.py
-3) recompute sky brightness files with rubin_sim.skybrightness.recalc_mags
-4) remake skybrightness_pre files with rubin_sim/rubin_sim/skybrightness_pre/data/generate_hdf5.py
-5) remake dark sky map with rubin_sim/rubin_sim/skybrightness_pre/data/generate_dark_sky.py
-6) tar and update files at SDF (throughputs, skybrightness, skybrightness_pre)
+0) update the throughputs in syseng_throughputs (this should be the original trigger to update throughputs anywhere downstream)
+1) update the throughputs in lsst/throughputs (including new tag)
+2) update rubin_sim_data/throughputs data files
+3) update rubin_scheduler.utils.sys_eng_vals.py - there is a notebook in syseng_throughputs which generates this file 
+4) recompute sky brightness files with rubin_sim.skybrightness.recalc_mags
+5) remake skybrightness_pre files with rubin_sim/rubin_sim/skybrightness_pre/data/generate_hdf5.py
+6) remake dark sky map with rubin_sim/rubin_sim/skybrightness_pre/data/generate_dark_sky.py
+7) tar and update files at SDF (throughputs, skybrightness, skybrightness_pre)