Initial commit

COPYRIGHT

@@ -0,0 +1 @@
+Copyright 2018 University of Illinois Board of Trustees

README.rst

@@ -0,0 +1,112 @@
+#########
+ctrl_oods
+#########
+
+``ctrl_oods`` is a package in the `LSST Science Pipelines <https://pipelines.lsst.io>`_.
+
+.. Add a brief (few sentence) description of what this package provides.
+
+The Observatory Operations Data Service watches for files in one or more directories, and then ingests them into an LSST Butler repository.   Files are expired from the repository at specified intervals.
+
+Usage:  oods.py [[-c config][-y config] [-v]
+
+Options:
+
+.. code:: bash
+
+ -c config use specified OODS YAML configuration file
+ -y config validate YAML configuration file
+ -v give verbose output
+
+Set up and usage
+----------------
+
+1) create the Gen2 Butler repository:
+
+.. code:: bash
+
+ mkdir repo
+ echo "lsst.obs.lsst.auxTel.AuxTelMapper" > repo/_mapper
+
+2) Edit the YAML configuration file.  The default example is located in:
+
+.. code:: bash
+
+    $CTRL_OODS_DIR/etc/oods.yaml
+
+3) Run the OODS:
+
+.. code:: bash
+
+    $CTRL_OODS_DIR/oods.py -c oods.yaml
+
+NOTE:  if you run the OODS without modifying the directory paths, it expects to scan for files in the directory in which the OODS has been invoked.   It will scan the directory "data" and use the Gen2 Butler repository "repo" by default.
+
+Configuration
+-------------
+
+The OODS is configured with the following YAML file:
+
+.. code:: yaml
+
+ defaultInterval: &interval
+     days: 0
+     hours: 0
+     minutes: 0
+     seconds: 0
+ 
+ ingester:
+     directories:
+         - data
+     butler:
+         class:
+             import : lsst.ctrl.oods.gen2ButlerIngester
+             name : Gen2ButlerIngester
+         repoDirectory : repo
+     batchSize: 20
+     scanInterval:
+         <<: \*interval
+         seconds: 10
+ 
+ cacheCleaner:
+     directories:
+         - repo/raw
+     scanInterval:
+         <<: \*interval
+         seconds: 30
+     filesOlderThan:
+         <<: \*interval
+         days: 30
+     directoriesEmptyForMoreThan:
+         <<: \*interval
+         days: 1
+
+The "defaultInterval" block is used as shorthand for the intervals used throughout the rest of the YAML configuration.
+
+The "ingester" block
+--------------------
+
+This has four sections:  directories, butler, scanInterval, and batchSize.
+
+The "directories" section takes a list of directories to watch.   By default this watches the "data" directory which is expected to be in the same directory in which the OODS runs.
+
+The "butler" section specified which type of LSST Butler to run, and the repository to use.  By default, it uses an object called Gen2ButlerIngester, specified by the import "lsst.ctrl.oods.gen2ButlerIngester".   If you write your own ingestion object, follow the pattern specified in this file.   By default the "repo" directory is expected to be in the same directory which the OODS runs.  This butler repository is expected to be set up properly (see below) before the first invocation of the OODS.
+
+The "scanInterval" section specifies the frequency at which to scan the "directories" specified above.  In the example, it scans every 10 seconds.
+
+The "batchSize" is set to the number of files to attempt to ingest at once.  The current version (0.1) of the OODS calls the obs_lsst package's "ingestImages.py" script, and it is possible to overload the command line beyond it's limit if too many files are specified on the command line at one time.  To prevent this, files are ingested in batches of "batchSize" or less.   Note that all files that are found when an ingestion requested at that "scanInterval" will attempt to be ingested.  Also note that in future versions, (Gen3 Butler), there will be
+a programmatic interface to the butler ingestion code, so this parameter will 
+likely be deprecated.
+
+The "cacheCleaner" block
+------------------------
+
+This has four sections: directories, scanInterval, filesOlderThan and directoriesEmptyForMoreThan.
+
+The "directories" section specifies the location of the ingested Butler files to clean up.   By default this is "repo/raw" and is expected to be in the same directory in which the OODS runs.
+
+The "scanInterval" section specifies the frequency at which to scan the "directories" specified above.  In the example, it scans every 30 seconds.
+
+The "filesOlderthan" section specifies how old files must be in order for them to be considered for removal.   This is checked against the last modification date of the file.  In this example, the file must be at least 30 days old to be considered for removal.
+
+The "directoriesEmptyForMoreThan" section specifies how long directories must be empty for before they are to be considered for removal.   This is checked against the last modification date of the directory.  In this example, the directory must be at least  1 day old and empty to be considered for removal.

SConstruct

@@ -0,0 +1,4 @@
+# -*- python -*-
+from lsst.sconsUtils import scripts
+# Python-only package
+scripts.BasicSConstruct("ctrl_oods", disableCc=True)

bin.src/SConscript

@@ -0,0 +1,3 @@
+# -*- python -*-
+from lsst.sconsUtils import scripts
+scripts.BasicSConscript.shebang()

bin.src/oods.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+
+#
+# LSST Data Management System
+#
+# Copyright 2008-2019  AURA/LSST.
+#
+# This product includes software developed by the
+# LSST Project (http://www.lsst.org/).
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the LSST License Statement and
+# the GNU General Public License along with this program.  If not,
+# see <https://www.lsstcorp.org/LegalNotices/>.
+#
+
+import optparse
+import os
+import sys
+import yaml
+import lsst.utils
+from lsst.ctrl.oods.taskRunner import TaskRunner
+from lsst.ctrl.oods.fileIngester import FileIngester
+from lsst.ctrl.oods.cacheCleaner import CacheCleaner
+from lsst.ctrl.oods.validator import Validator
+
+usage = """usage: %prog [[-c config]|[-y config]]"""
+
+parser = optparse.OptionParser("usage")
+parser.add_option("-c", "--config", action="store", dest="configFile",
+                  default=None, help="OODS configuration file")
+parser.add_option("-y", "--yaml-validate", action="store_true", dest="validate",
+                  default=False, help="validate YAML file")
+parser.add_option("-v", "--verbose", action="store_true",
+                  dest="verbose", default=False, help="verbose output")
+
+parser.opts = {}
+parser.args = []
+
+(parser.opts, parser.args) = parser.parse_args()
+
+if parser.opts.validate is True:
+    with open(parser.args[0], 'r') as f:
+        config = yaml.load(f)
+        v = Validator(config, parser.opts.verbose)
+        v.verify()
+        if not v.isValid:
+            print("invalid OODS YAML configuration file")
+        else:
+            print("valid OODS YAML configuration file")
+    sys.exit(0)
+
+package = lsst.utils.getPackageDir("ctrl_oods")
+yamlPath = os.path.join(package, "etc", "oods.yaml")
+if parser.opts.configFile is not None:
+    yamlPath = os.path.join(parser.opts.configFile)
+
+oodsConfig = None
+with open(yamlPath, 'r') as f:
+    oodsConfig = yaml.load(f)
+
+
+print("starting...")
+
+
+ingesterConfig = oodsConfig["ingester"]
+ingester = FileIngester(ingesterConfig, parser.args.verbose)
+ingest = TaskRunner(interval=ingesterConfig["scanInterval"],
+                    task=ingester.runTask)
+
+cacheConfig = oodsConfig["cacheCleaner"]
+cacheCleaner = CacheCleaner(cacheConfig, parser.args.verbose)
+cleaner = TaskRunner(interval=cacheConfig["scanInterval"],
+                     task=cacheCleaner.runTask)
+
+ingest.start()
+cleaner.start()
+
+ingest.join()
+cleaner.join()

doc/.gitignore

@@ -0,0 +1,10 @@
+# Doxygen products
+html
+xml
+*.tag
+*.inc
+doxygen.conf
+
+# Sphinx products
+_build
+py-api

doc/conf.py

@@ -0,0 +1,13 @@
+"""Sphinx configuration file for an LSST stack package.
+
+This configuration only affects single-package Sphinx documentation builds.
+"""
+
+from documenteer.sphinxconfig.stackconf import build_package_configs
+import lsst.ctrl.oods
+
+
+_g = globals()
+_g.update(build_package_configs(
+    project_name='ctrl_oods',
+    version=lsst.ctrl.oods.version.__version__))

doc/index.rst

@@ -0,0 +1,12 @@
+###############################
+ctrl_oods documentation preview
+###############################
+
+.. This page is for local development only. It isn't published to pipelines.lsst.io.
+
+.. Link the index pages of package and module documentation directions (listed in manifest.yaml).
+
+.. toctree::
+   :maxdepth: 1
+
+   lsst.ctrl.oods/index

doc/lsst.ctrl.oods/index.rst

@@ -0,0 +1,41 @@
+.. py:currentmodule:: lsst.ctrl.oods
+
+.. _lsst.ctrl.oods:
+
+##############
+lsst.ctrl.oods
+##############
+
+.. Paragraph that describes what this Python module does and links to related modules and frameworks.
+
+.. .. _lsst.ctrl.oods-using:
+
+.. Using lsst.ctrl.oods
+.. ====================
+
+.. toctree linking to topics related to using the module's APIs.
+
+.. .. toctree::
+..    :maxdepth: 1
+
+.. _lsst.ctrl.oods-contributing:
+
+Contributing
+============
+
+``lsst.ctrl.oods`` is developed at https://github.com/lsst-dm/ctrl_oods.
+You can find Jira issues for this module under the `ctrl_oods <https://jira.lsstcorp.org/issues/?jql=project%20%3D%20DM%20AND%20component%20%3D%20ctrl_oods>`_ component.
+
+.. If there are topics related to developing this module (rather than using it), link to this from a toctree placed here.
+
+.. .. toctree::
+..    :maxdepth: 1
+
+.. _lsst.ctrl.oods-pyapi:
+
+Python API reference
+====================
+
+.. automodapi:: lsst.ctrl.oods
+   :no-main-docstr:
+   :no-inheritance-diagram:

doc/manifest.yaml

@@ -0,0 +1,12 @@
+# Documentation manifest.
+
+# List of names of Python modules in this package.
+# For each module there is a corresponding module doc subdirectory.
+modules:
+  - "lsst.ctrl.oods"
+
+# Name of the static content directories (subdirectories of `_static`).
+# Static content directories are usually named after the package.
+# Most packages do not need a static content directory (leave commented out).
+# statics:
+#   - "_static/ctrl_oods"

etc/oods.yaml

@@ -0,0 +1,31 @@
+defaultInterval: &interval
+    days: 0
+    hours: 0
+    minutes: 0
+    seconds: 0
+
+ingester:
+    directories:
+        - data
+    butler:
+        class:
+            import : lsst.ctrl.oods.gen2ButlerIngester
+            name : Gen2ButlerIngester
+        repoDirectory : repo
+    batchSize: 20
+    scanInterval:
+        <<: *interval
+        seconds: 10
+
+cacheCleaner:
+    directories:
+        - repo/raw
+    scanInterval:
+        <<: *interval
+        seconds: 30
+    filesOlderThan:
+        <<: *interval
+        days: 30 
+    directoriesEmptyForMoreThan:
+        <<: *interval
+        days: 1

examples/SConscript

@@ -0,0 +1,3 @@
+# -*- python -*-
+from lsst.sconsUtils import scripts
+scripts.BasicSConscript.examples()

python/lsst/__init__.py

@@ -0,0 +1,23 @@
+# This file is part of ctrl_oods.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (https://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import pkgutil, lsstimport
+__path__ = pkgutil.extend_path(__path__, __name__)

python/lsst/ctrl/__init__.py

@@ -0,0 +1,23 @@
+# This file is part of ctrl_oods.
+#
+# Developed for the LSST Data Management System.
+# This product includes software developed by the LSST Project
+# (https://www.lsst.org).
+# See the COPYRIGHT file at the top-level directory of this distribution
+# for details of code ownership.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import pkgutil, lsstimport
+__path__ = pkgutil.extend_path(__path__, __name__)