Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure mode of pipeline: Empty filename #33

Open
laldoroty opened this issue Oct 21, 2024 · 2 comments
Open

Failure mode of pipeline: Empty filename #33

laldoroty opened this issue Oct 21, 2024 · 2 comments

Comments

@laldoroty
Copy link
Collaborator

Some files are failing like this:

The following have been reloaded with a version change:
  1) conda/Mambaforge-23.1.0-1 => conda/Miniconda3-py311_23.11.0-2


The following have been reloaded with a version change:
  1) conda/Miniconda3-py311_23.11.0-2 => conda/Mambaforge-23.1.0-1

20000677
[2024-10-21 13:44:55 - phrosty - ERROR] Sky subtraction subprocess failure: Empty or corrupt FITS file

MeLOn CheckPoint [Roman_TDS_simple_model_H158_53350_11.fits]: Run Python Wrapper of SExtractor!
WARNING: File may have been truncated: actual file length (1458176) is smaller than the expected size (133701120) [astropy.io.fits.file]
[2024-10-21 13:44:56 - phrosty - ERROR] Sky subtraction subprocess failure: list index out of range

MeLOn CheckPoint [Roman_TDS_simple_model_H158_7585_1.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_53735_8.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_609_12.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_7580_8.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_614_17.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_4509_13.fits]: Run Python Wrapper of SExtractor!

MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: Run Python Wrapper of SExtractor!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53350_11.fits]: SExtractor found [3789] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53350_11.fits]: PYSEx output catalog contains [3789] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_7585_1.fits]: SExtractor found [4312] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_7585_1.fits]: PYSEx output catalog contains [4312] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: SExtractor found [3899] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: PYSEx output catalog contains [3899] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53735_8.fits]: SExtractor found [3903] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53735_8.fits]: PYSEx output catalog contains [3903] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_609_12.fits]: SExtractor found [3945] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_609_12.fits]: PYSEx output catalog contains [3945] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_7580_8.fits]: SExtractor found [3929] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_7580_8.fits]: PYSEx output catalog contains [3929] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_614_17.fits]: SExtractor found [4056] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_614_17.fits]: PYSEx output catalog contains [4056] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_4509_13.fits]: SExtractor found [4240] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_4509_13.fits]: PYSEx output catalog contains [4240] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: SExtractor found [3899] sources!
MeLOn CheckPoint [Roman_TDS_simple_model_H158_53355_6.fits]: PYSEx output catalog contains [3899] sources!
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/erfa/core.py:133: ErfaWarning: ERFA function "d2dtf" yielded 1 of "dubious year (Note 5)"
  warn(f'ERFA function "{func_name}" yielded {wmsg}', ErfaWarning)
MeLOn CheckPoint: Cuda resampling takes [0.004495 s]
MeLOn CheckPoint: Cuda resampling takes [0.000037 s]
MeLOn CheckPoint: Cuda resampling takes [0.000122 s]
MeLOn CheckPoint: TRIGGER Function Compilations of SFFT-SUBTRACTION!

 --//--//--//--//-- TRIGGER SFFT COMPILATION --//--//--//--//-- 

 ---//--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---//--- 

 --//--//--//--//-- EXIT SFFT COMPILATION --//--//--//--//-- 

MeLOn Report: Function Compilations of SFFT-SUBTRACTION TAKES [0.001 s]

                                __    __    __    __
                               /  \  /  \  /  \  /  \
                              /    \/    \/    \/    \
            █████████████████/  /██/  /██/  /██/  /█████████████████████████
                            /  / \   / \   / \   / \  \____
                           /  /   \_/   \_/   \_/   \    o \__,
                          / _/                       \_____/  `
                          |/
        
                      █████████  ███████████ ███████████ ███████████        
                     ███░░░░░███░░███░░░░░░█░░███░░░░░░█░█░░░███░░░█            
                    ░███    ░░░  ░███   █ ░  ░███   █ ░ ░   ░███  ░ 
                    ░░█████████  ░███████    ░███████       ░███    
                     ░░░░░░░░███ ░███░░░█    ░███░░░█       ░███    
                     ███    ░███ ░███  ░     ░███  ░        ░███    
                    ░░█████████  █████       █████          █████   
                     ░░░░░░░░░  ░░░░░       ░░░░░          ░░░░░         
        
                    Saccadic Fast Fourier Transform (SFFT) algorithm
                    sfft supported by @LeiHu
        
                    GitHub: https://github.com/thomasvrussell/sfft
                    Related Paper: https://arxiv.org/abs/2109.09334
                    
            ████████████████████████████████████████████████████████████████
            
            

 --||--||--||--||-- TRIGGER SFFT SUBTRACTION --||--||--||--||-- 

 ---||--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---||--- 

MeLOn CheckPoint: SFFT-SUBTRACTION Preliminary Steps takes [0.0612s]
/////   a   ///// Read Input Images  (0.0000s)
/////   b   ///// Spatial Polynomial (0.0090s)
/////   c   ///// DFT-8             (0.0499s)

MeLOn CheckPoint: SFFT-SUBTRACTION Establish & Solve Linear System takes [0.4114s]
/////   d   ///// Establish OMG                       (0.1928s)
/////   e   ///// Establish GAM                       (0.0412s)
/////   f   ///// Establish PSI                       (0.0339s)
/////   g   ///// Establish PHI                       (0.0150s)
/////   h   ///// Establish THE & DEL                 (0.0473s)
/////   i   ///// Solve Linear System                 (0.0725s)

 --||--||--||--||-- EXIT SFFT SUBTRACTION --||--||--||--||-- 

 --||--||--||--||-- TRIGGER SFFT SUBTRACTION --||--||--||--||-- 

 ---||--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---||--- 

MeLOn CheckPoint: SFFT-SUBTRACTION Preliminary Steps takes [0.0028s]
/////   a   ///// Read Input Images  (0.0000s)
/////   b   ///// Spatial Polynomial (0.0001s)
/////   c   ///// DFT-8             (0.0010s)

MeLOn CheckPoint: SFFT-SUBTRACTION Perform Subtraction takes [0.2554s]
/////   j   ///// Calculate Kab         (0.0657s)
/////   k   ///// Construct DIFF        (0.1897s)

 --||--||--||--||-- EXIT SFFT SUBTRACTION --||--||--||--||-- 

MeLOn Report: SFFT-SUBTRACTION TAKES [0.790 s]
MeLOn CheckPoint: Cuda resampling takes [0.000052 s]
MeLOn CheckPoint: Cuda resampling takes [0.000038 s]
MeLOn CheckPoint: Cuda resampling takes [0.000052 s]
MeLOn CheckPoint: TRIGGER Function Compilations of SFFT-SUBTRACTION!

 --//--//--//--//-- TRIGGER SFFT COMPILATION --//--//--//--//-- 

 ---//--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---//--- 

 --//--//--//--//-- EXIT SFFT COMPILATION --//--//--//--//-- 

MeLOn Report: Function Compilations of SFFT-SUBTRACTION TAKES [0.002 s]

                                __    __    __    __
                               /  \  /  \  /  \  /  \
                              /    \/    \/    \/    \
            █████████████████/  /██/  /██/  /██/  /█████████████████████████
                            /  / \   / \   / \   / \  \____
                           /  /   \_/   \_/   \_/   \    o \__,
                          / _/                       \_____/  `
                          |/
        
                      █████████  ███████████ ███████████ ███████████        
                     ███░░░░░███░░███░░░░░░█░░███░░░░░░█░█░░░███░░░█            
                    ░███    ░░░  ░███   █ ░  ░███   █ ░ ░   ░███  ░ 
                    ░░█████████  ░███████    ░███████       ░███    
                     ░░░░░░░░███ ░███░░░█    ░███░░░█       ░███    
                     ███    ░███ ░███  ░     ░███  ░        ░███    
                    ░░█████████  █████       █████          █████   
                     ░░░░░░░░░  ░░░░░       ░░░░░          ░░░░░         
        
                    Saccadic Fast Fourier Transform (SFFT) algorithm
                    sfft supported by @LeiHu
        
                    GitHub: https://github.com/thomasvrussell/sfft
                    Related Paper: https://arxiv.org/abs/2109.09334
                    
            ████████████████████████████████████████████████████████████████
            
            

 --||--||--||--||-- TRIGGER SFFT SUBTRACTION --||--||--||--||-- 

 ---||--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---||--- 

MeLOn CheckPoint: SFFT-SUBTRACTION Preliminary Steps takes [0.0031s]
/////   a   ///// Read Input Images  (0.0000s)
/////   b   ///// Spatial Polynomial (0.0001s)
/////   c   ///// DFT-8             (0.0010s)

MeLOn CheckPoint: SFFT-SUBTRACTION Establish & Solve Linear System takes [0.3452s]
/////   d   ///// Establish OMG                       (0.0028s)
/////   e   ///// Establish GAM                       (0.0005s)
/////   f   ///// Establish PSI                       (0.0005s)
/////   g   ///// Establish PHI                       (0.0002s)
/////   h   ///// Establish THE & DEL                 (0.0008s)
/////   i   ///// Solve Linear System                 (0.3402s)

 --||--||--||--||-- EXIT SFFT SUBTRACTION --||--||--||--||-- 

 --||--||--||--||-- TRIGGER SFFT SUBTRACTION --||--||--||--||-- 

 ---||--- KerPolyOrder 2 | BGPolyOrder 0 | KerHW [9] ---||--- 

MeLOn CheckPoint: SFFT-SUBTRACTION Preliminary Steps takes [0.0027s]
/////   a   ///// Read Input Images  (0.0000s)
/////   b   ///// Spatial Polynomial (0.0001s)
/////   c   ///// DFT-8             (0.0010s)

MeLOn CheckPoint: SFFT-SUBTRACTION Perform Subtraction takes [0.0021s]
/////   j   ///// Calculate Kab         (0.0019s)
/////   k   ///// Construct DIFF        (0.0002s)

 --||--||--||--||-- EXIT SFFT SUBTRACTION --||--||--||--||-- 

MeLOn Report: SFFT-SUBTRACTION TAKES [0.361 s]
Traceback (most recent call last):
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 559, in <module>
    main()
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 555, in main
    pipeline( args.through_step )
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 465, in __call__
    sfftifier = self.align_and_pre_convolve( templ_image, sci_image )
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 180, in align_and_pre_convolve
    with fits.open( sci_image.skysub_path ) as hdul:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 220, in fitsopen
    raise ValueError(f"Empty filename: {name!r}")
ValueError: Empty filename: None

Should track this down.
@laldoroty
Copy link
Collaborator Author

@rknop can you write a brief summary of the solution and close this?

@laldoroty laldoroty changed the title Failure mode of pipeline Failure mode of pipeline: Empty filename Oct 22, 2024
@laldoroty
Copy link
Collaborator Author

@rknop what was the solution to this? I can't recall, but my previous comment indicates that you fixed it at some point...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@laldoroty