end game tests

[beckermr]

• get setup scripts together for erin (matt)
• test variable weight map (matt)
• write cosmic+bad column generator for montara and test (sid?)
• get things running on condor (erin)

[esheldon]

Should we commit an environment file for conda as well?

[beckermr]

yes when we have the final config - here is a draft one: https://www.cosmo.bnl.gov/Private/gpfs/workarea/beckermr/des-y6-imsims-env.yaml

[beckermr]

If you make this env, then a script to run the sims looks like

#!/bin/bash

export IMSIM_DATA=${MEDS_DIR}
export TMPDIR=/data/beckermr/tmp

mkdir -p ./sim_outputs_plus_$1_$2
run-eastlake-sim \
  -v 1 \
  --seed $2 \
  eastlake-config.yaml \
  ./sim_outputs_plus_$1_$2 stamp.shear.g1=0.02 stamp.shear.g2=0.0 gal.flux.fgmag=$1

mkdir -p ./sim_outputs_minus_$1_$2
run-eastlake-sim \
  -v 1 \
  --seed $2 \
  eastlake-config.yaml \
  ./sim_outputs_minus_$1_$2 stamp.shear.g1=-0.02 stamp.shear.g2=0.0 gal.flux.fgmag=$1

  # --step_names pizza_cutter metadetect \
  # --resume_from=./sim_outputs/job_record.pkl \

  # --skip_completed_steps \
  # --resume_from=./sim_outputs/job_record.pkl \

#  --step_names src_extractor \
#  --resume_from=./sim_outputs/job_record.pkl \

you can use my MEDS_DIR at BNL: https://www.cosmo.bnl.gov/Private/gpfs/workarea/beckermr/MEDS_DIR/

[beckermr]

You'll also need the various yaml configs here: https://www.cosmo.bnl.gov/Private/gpfs/workarea/beckermr/des-y6-analysis/2023_05_09_imsims_noise_debug/

[esheldon]

I see this warning, is it ok?

Warning: config dict contains the following unexpected fields: ['pipeline', 'galsim_montara', 'true_detection', 'pizza_cutter', 'metadetect', 'star'].

[esheldon]

I edited the eastlake config where it had number of jobs n_jobs set for both pizza cutter and metadetect. I changed them to 1 but it is using all the cores.

[esheldon]

I think I see the problem: the part run before pizza cutter is using all cores. How do I set that to 1 core?

[beckermr]

That error is fine.

Look in the output section. There should be something to control how many cores are used there.

[esheldon]

That worked, thanks.

The field is "nproc"

[esheldon]

I ran with badpixfromfits in the config

  # uses the bit mask from the real image                                                                                                                                                                                                                      
  badpixfromfits:                                                                                                                                                                                                                                              
    hdu: 1                                                                                                                                                                                                                                                     
    mask_hdu: 2                                                                                                                                                                                                                                                
    mask_file: "$orig_image_path"

and am now getting this

Start file 0 = /gpfs/mnt/gpfs02/astro/desdata/esheldon/lensing/des-lensing/y6-sims/mysims/mysim_outputs_plus_18_1234/des-pizza-slices-y6-v15-nostars/DES0433-2332/sources-r/OPS_Taiga/finalcut/Y5A1/r3518/20180107/D00710310/p01/red/immask/D00710310_r_c34_r35
18p01_immasked.fits
Start file 1 = /gpfs/mnt/gpfs02/astro/desdata/esheldon/lensing/des-lensing/y6-sims/mysims/mysim_outputs_plus_18_1234/des-pizza-slices-y6-v15-nostars/DES0433-2332/sources-r/OPS_Taiga/finalcut/Y5A1/r3567/20141216/D00388613/p02/red/immask/D00388613_r_c55_r35
67p02_immasked.fits
adding bkg with mean 0.00e+00 from file /gpfs/mnt/gpfs02/astro/desdata/esheldon/lensing/des-lensing/y6-sims/mysims/mysim_outputs_plus_18_1234/des-pizza-slices-y6-v15-nostars/DES0433-2332/sources-r/OPS_Taiga/finalcut/Y5A1/r3518/20180107/D00710310/p01/red/b
kg/D00710310_r_c34_r3518p01_bkg.fits.fz
Process-1: Exception caught for file 0 = /gpfs/mnt/gpfs02/astro/desdata/esheldon/lensing/des-lensing/y6-sims/mysims/mysim_outputs_plus_18_1234/des-pizza-slices-y6-v15-nostars/DES0433-2332/sources-r/OPS_Taiga/finalcut/Y5A1/r3518/20180107/D00710310/p01/red/
immask/D00710310_r_c34_r3518p01_immasked.fits
Traceback (most recent call last):
  File "/gpfs/mnt/gpfs02/astro/desdata/esheldon/mambaforge/envs/y6sims/lib/python3.10/site-packages/galsim/config/util.py", line 734, in worker
    result = job_func(**kwargs)
  File "/gpfs/mnt/gpfs02/astro/desdata/esheldon/mambaforge/envs/y6sims/lib/python3.10/site-packages/galsim/config/output.py", line 260, in BuildFile
    data = builder.addExtraOutputHDUs(config, data, logger)
  File "/gpfs/mnt/gpfs02/astro/desdata/esheldon/mambaforge/envs/y6sims/lib/python3.10/site-packages/galsim/config/output.py", line 546, in addExtraOutputHDUs
    data = AddExtraOutputHDUs(config, data, logger)
  File "/gpfs/mnt/gpfs02/astro/desdata/esheldon/mambaforge/envs/y6sims/lib/python3.10/site-packages/galsim/config/extra.py", line 284, in AddExtraOutputHDUs
    raise GalSimConfigValueError("hdu is invalid or a duplicate.",hdu)
galsim.errors.GalSimConfigValueError: hdu is invalid or a duplicate. Value 1

[beckermr]

Did you comment out the other pixel map section?

[esheldon]

sorry which part do you mean?

[esheldon]

I don't see a section like "pixel_map"

[beckermr]

I mean the other section that is badpix

[esheldon]

thanks, that seems to work