Bug preventing using custom cache without state flag #126
Comments
areeda
pushed a commit
to areeda/pyomicron
that referenced
this issue
Sep 6, 2022
…hout accessing dqsegdb. Add documentation for this
areeda
pushed a commit
to areeda/pyomicron
that referenced
this issue
Sep 9, 2022
…ache without accessing dqsegdb. Add documentation for this" I inadvertently made changes after PR was approved This reverts commit ddbe108.
areeda
added a commit
that referenced
this issue
Sep 9, 2022
) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Revert "Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this" I inadvertently made changes after PR was approved This reverts commit ddbe108. * Minor flake8 complaint Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Sep 27, 2022
* Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
to areeda/pyomicron
that referenced
this issue
Sep 27, 2022
* Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Oct 16, 2022
* Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
to areeda/pyomicron
that referenced
this issue
Nov 22, 2022
* Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
to areeda/pyomicron
that referenced
this issue
Nov 22, 2022
* Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (gwpy#146) * Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
to areeda/pyomicron
that referenced
this issue
Nov 22, 2022
* Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Mar 28, 2023
* Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
to areeda/pyomicron
that referenced
this issue
Apr 5, 2023
* Do not cross metric day boundaries. * Merge day boundary (gwpy#146) * Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (gwpy#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (gwpy#146) * Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (gwpy#146) * Address issue gwpy#126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Dec 4, 2023
* Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Fix a double gzip of ligolw files (#151) * Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <[email protected]> * Resolve flake8 issues * Update log format to use human readble date/time instead of gps tweak logging to better underst guardian channel usage- * remove old setup.cfg * Work vonpytest failures. The remaining errors are the result of omicron segfaults if environment variable not set * missing blank line, from flake8 * Update condor defaults in argparse. Use local time in logs not gps * uncomment test I had ignored because of omicron segfaults * remove extra blank lines * Make sure we have a maximum segment for online processing, default = 1200 seconds. * flake8 on github is different than my workstation --------- Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Dec 12, 2023
* Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Fix a double gzip of ligolw files (#151) * Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <[email protected]> * Resolve flake8 issues * Update log format to use human readble date/time instead of gps tweak logging to better underst guardian channel usage- * remove old setup.cfg * Work vonpytest failures. The remaining errors are the result of omicron segfaults if environment variable not set * missing blank line, from flake8 * Up the default max-concurrent to 64 in hopes of much higher throughput. And deal with it properly instead of my paren/child hack. --------- Co-authored-by: Joseph Areeda <[email protected]>
areeda
added a commit
that referenced
this issue
Sep 25, 2024
* Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Fix a double gzip of ligolw files (#151) * Do not cross metric day boundaries. * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * Check point merge (#147) * Do not cross metric day boundaries. * add log file arg delete empty directories when done * Tweak remove empty dir removal * Tweak remove empty dir removal again * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * rebase agaist last approved PR * rebase against last approved PR * rebase against last approved PR again, fix flake8 * Fix a bug in remove empty directories. Co-authored-by: Joseph Areeda <[email protected]> * Merge day boundary (#146) * Address issue #126. Allow pyomicron to run from a frame cache without accessing dqsegdb. Add documentation for this * Do not merge files if they overlap "metric days" * Do not cross metric day boundaries. Co-authored-by: Joseph Areeda <[email protected]> * minor doc changes * Fix a bug where an xml.gz file could get compressed again in merge-with-gaps * Implement a periodic vacate to address permanent D-state (uninterupptible wait) causing jobs to fail to complete * Always create a log file. If not specified put one in the output directory * Fix a problem with periodic vacate. * Up the periodic vacate time to 3 hrs * Found a job killing typo * Add time limits to post processing also * Don't save segments.txt file if no sgments founds because we don't know if it's an issue of not finding them or a valid not analyzable state. * disable periodic vacate to demo the problem. * Fix reported version in some utilities. Only update segments.txt if omicron is actually run. * Clarify relative imports. and add details to a few log messages * Resolve flake8 issues --------- Co-authored-by: Joseph Areeda <[email protected]> * Resolve flake8 issues * Update log format to use human readble date/time instead of gps tweak logging to better underst guardian channel usage- * remove old setup.cfg * Work vonpytest failures. The remaining errors are the result of omicron segfaults if environment variable not set * missing blank line, from flake8 * Fix a problem wit hmax-online-lookback not working properly in all pathes. Add some uman readable date/times to gps messages * Fix logging problems from different gps time objects * Better logging why online effort did not run * Up default lookback window to 40 min * Up default maximum lookback window to 60 min. Better logging of why we did not run. * Fix flake8 and more logging updates * Fix flake8 and more logging updates * More logging updates * More logging updates, paths through could cause error * Trap and print errors from main() * fix dag submission command * add smart postscript to allow retries befor ignoring errors * tst version of scitokens and smart post script * tst version of scitokens and smart post script * add arg to specify auth type (x5099, vault or apissuer) * memopry units in the wrong place * memory units in the wrong place, condor_run * flake8 nit picked * Again try to get periodic_release and periodic_remove correct * Sort console scripts * Typo in periodic_remove * Better error message when programs not available * implement cona run for all jobs in dag * conda run complications with cvmfs * archive.py deals with renamed trigger files * archive.py deals with renamed trigger files take 2 * condor run needed in all scripts. * minor logging changes * working on archive issues * working on archive issues, keep "temporary" files to help debugging * more logging * Default max lookback changed to 30min. more logging tweaks * Add omicron_utils to insta;; requirements * Work on build and test workflow error. * Still working on build and test workflow error. Remove Python 3.9 from workflows * Set loglevel for OmicronConfig to Critical so --version command is clean * Resolve all conversations * try to deal with github error ``` Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v2`. Learn more: https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/ ``` * try to deal with github error ``` Error: This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v2`. Learn more: https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/ ``` --------- Co-authored-by: Joseph Areeda <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a cache file is provided, but no state flag is specified, the analyzable segments are calculated using this function call:
pyomicron/omicron/cli/process.py
Line 708 in 42c2a51
Since no frame type is needed when using a cache file, this returns an empty segment list. This then results in a runtime error at the following line:
pyomicron/omicron/cli/process.py
Line 829 in 42c2a51
This can be avoided by providing a state flag to calculate the analyzable segments. However, if one is attempting to analyze user-generated data, a usable state flag may not exist. The ideal behavior when using a cache file (and no state flag) is to use the cache file itself to generate the analyzable segments.
This can be fixed by adding an extra
elifstatement for this situation, which usesgwpy.io.cache.cache_segments()to calculate the required segments.The text was updated successfully, but these errors were encountered: