CacheProdV6

Cache production v6 instructions

Preparation

1. Prepare cmssw_env.sh to run in Pisa grid environment

   export SCRAM_ARCH=slc7_amd64_gcc820
   export VO_CMS_SW_DIR=/cvmfs/cms.cern.ch/
   source $VO_CMS_SW_DIR/cmsset_default.sh
   export PATH=$PATH:/home/users/androsov/soft/cmake/3.17.1/bin
   export PATH=$(echo -n $PATH | awk -v RS=: '!($0 in a) {a[$0]; printf("%s%s", length(a) > 1 ? ":" : "", $0)}')

2. Connect to fai machine using CentOS7 image:

   bsub -Is -n 4 -q fai -a docker-cs7 /bin/zsh

3. Install framework

   curl -s https://raw.githubusercontent.com/hh-italian-group/h-tautau/master/install_framework.sh | bash -s ana

4. Compile CacheTupleProducer and CacheMerger

   cd CMSSW_11_1_0_pre6/src
   cmsenv
   ./run.sh --make CacheTupleProducer
   ./run.sh --make CacheMerger

Submit jobs

The following command line should be used to submit and control jobs:

UNC=XXX; YEAR=XXX; VER=6; python -u ./h-tautau/Instruments/python/submit_cache_producer.py --input /gpfs/ddn/cms/user/androsov/store/cms-it-hh-bbtautau/Tuples${YEAR}_v$VER/Full --output output/Tuples${YEAR}_v$VER/$UNC --central-output /gpfs/ddn/cms/user/androsov/store/cms-it-hh-bbtautau/Tuples${YEAR}_v$VER/Cache/$UNC --pattern '.*\.root' --unc-sources $UNC --period Run${YEAR} --queue cms --data-file-pattern '.*Run201[678][A-H].*' --long-jobs /gpfs/ddn/cms/user/androsov/store/cms-it-hh-bbtautau/Tuples${YEAR}_v$VER/Cache/long_jobs.txt 2>&1 | tee status_${YEAR}_v${VER}_${UNC}_$(date +"%Y-%m-%dT%H%M").log

where UNC and YEAR values should be changing accordingly.

Long jobs

For the first UNC run of the given year, long_jobs.txt file should be created manually. The recommended job duration is around 6 hours.

For the consecutive iterations, the execution time statistics collected during the previous round can be used to automatically create long_jobs.txt:

UNC=XXX; YEAR=XXX; VER=6; ./h-tautau/Instruments/python/export_cache_summary.py /gpfs/ddn/cms/user/androsov/store/cms-it-hh-bbtautau/Tuples${YEAR}_v${VER}/Cache/${UNC}/ > job_summary.csv
./h-tautau/Instruments/python/create_cache_splitting.py --input job_summary.csv --max-exe-time 6 > long_jobs.txt

Failed jobs

Once the reason that caused jobs to fail is understood and fixed, the output directories of the failed jobs should be removed using the information from the last status long:

UNC=XXX; YEAR=XXX; VER=6; LOG_FILE=$(ls -t status_${YEAR}_v${VER}_${UNC}_*.log | head -1); cat $LOG_FILE | grep -E ': failed' | sed -E 's/^(.*)_(eTau|muTau|tauTau)(|.*):.*/\2\/\1\3/' | xargs -n 1 -I JOB rm -r output/Tuples${YEAR}_v${VER}/$UNC/JOB

Uncertainty variations

1. None
2. TauES
3. JetReduced_Total
4. EleFakingTauES
5. MuFakingTauES
6. JetReduced_Absolute
7. JetReduced_Absolute_year
8. JetReduced_BBEC1
9. JetReduced_BBEC1_year
10. JetReduced_EC2
11. JetReduced_EC2_year
12. JetReduced_FlavorQCD
13. JetReduced_HF
14. JetReduced_HF_year
15. JetReduced_RelativeBal
16. JetReduced_RelativeSample_year
17. TauES_DM0
18. TauES_DM1
19. TauES_DM10
20. TauES_DM11
21. EleFakingTauES_DM0
22. EleFakingTauES_DM1