sort out confusion/duplicates in tarballs handling from TW to sched to Wrapper

[belforte]

currently TW sends one file to scheduler: InputFiles.tar.gz

• expanded by dag_bootstrap_startup.sh
• content in [*]
• contains the two code tarballs
  • CMSRunAnalysis.tar.gz (which is shipped to WN)
  • TaskManagerRun.tar.gz
• also contains the user sandbox.tar.gz

CMSRunAnalysis.tar.gz is expanded by CMSRunAnalysis.sh [1]. It has

• WMCore.zip
• CMSRunAnalysis.py, TweakPSet.py and 3 other py file which are likely not needed [2]

TaskManagerRun.tar.gz contains files used by scripts running on the scheduler [4]

• expanded by dagBoostrap.sh [3]
• contains files which are duplicates of what's in CMSRunAnalysis.tar.gz [4]

[*]

annotated content

Action Needed	file	purpose
N	gWMS-CMSRunAnalysis.sh	the JobWrapper. command to be passed to HTC in the JDL when submitting to global pool
N	submit_env.sh	used by job wrapper
N	CMSRunAnalysis.sh	by job wrapper to unpack CMSRunAnalysis.tar.gz
Y	cmscp.py	used by job wrapper (could be moved inside CMSRunAnalysis.tar.gz )
Y	cmscp.sh	used by job wrapper (could be moved inside CMSRunAnalysis.tar.gz)
Y	RunJobs.dag	the DAG to be run for this task. Also send via transfer_input_files ? REMOVE ?
N	Job.submit	template for the JDL when submitting to grid, will be customized in Job.1.submit etc.
N	dag_bootstrap.sh	used inside DAG to setup the env for PreJobs, PostJob, PreDag steps
N	AdjustSites.py	called by dag_bootstrap_startup.sh*
Y	site.ad	list of available sites for each job in classAd format (obsolete ?) manipulated by AdjustSites but never used ? REMOVE ?
Y	site.ad.json	same in JSON format. Used in PreJob.py. Not touched in AdjustSites ! SHOULD CHANGE ?
N	datadiscovery.pkl	for automatic splitting
N	taskinformation.pkl	for automatic splitting
N	taskworkerconfig.pkl	for automatic splitting
N	run_and_lumis.tar.gz	input runs and lumis for each job
N	input_files.tar.gz	input files for each job
Y	debug/	subdirectory contains the two files below, uploaded by CRABClient to S3. Can remove ?
Y	debug/crabConfig.py	user config file, useful debugging info
Y	debug/originalPSet.py	user PSet file, useful debugging info
Y	debug/	user ScriptExe file (if present), useful debugging info
N	sandbox.tar.gz	the user files, uploaded by CRABClient to S3
Y	CMSRunAnalysis.tar.gz	to be transferred to running job via HTC file transfer mechanism.
Y	TaskManagerRun.tar.gz
N	input_dataset_lumis.json	list of lumis in input dataset, for use by crab report
Y	input_dataset_duplicate_lumis.json	retrieved by crab report, but .... needed ?
N	debug_files.tar.gz	tarball whose contend has been expanded in debug/ . DUPLICATION
N	input_args.json	to be fetched by CRABClient and used in preparelocal

[1]

CRABServer/scripts/CMSRunAnalysis.sh

Line 49 in b3b26db

tar xmf CMSRunAnalysis.tar.gz || exit 10042

[2]
WMCore.zip
TweakPSet.py
CMSRunAnalysis.py
ServerUtilities.py Not needed
CMSGroupMapper.py Not needed
RESTInteractions.py Not needed

[3]

CRABServer/scripts/dag_bootstrap.sh

Line 69 in b3b26db

tar xvfm TaskManagerRun.tar.gz

[4]

CRAB3.zip
TweakPSet.py Not needed
CMSRunAnalysis.py Not needed
task_process/
task_process/CMSRucio.py
task_process/FTS_Transfers.py
task_process/RUCIO_Transfers.py
task_process/cache_status.py
task_process/task_proc_wrapper.sh
ServerUtilities.py
RucioUtils.py
CMSGroupMapper.py
RESTInteractions.py

[belforte]

fall out action items

• remove extra files from TaskManagerRun.tar.gz - Remove extra files from TW tarballs #8718
• remove extra files from CMSRunAnalysis.tar.gz - Remove extra files from TW tarballs #8718
  • uff. current cmscp.py imports parseJobAd from ServerUtilities but can use code from CMSRunAnalysislpy instead [1] ! Also there is no smv2-lcg any more, not need to check if gfal-copy exists - Remove extra files from TW tarballs #8718
• move cmscp.* to CMSRunAnalysis.tar.gz - postponed, maybe part of bigger restruct as per comment below ?
• verify if RunJobs.dag can be removed from InputFiles.tar.gz
  • NO. It can not.
• clarify situation with debug files and possibly remove debug/ directory from tarball (keep only debug_files.tar.gz) - moved to change how user "debug" files in debug_files.tar.gz are handled  #8719
• clarify site.ad vs. site.ad.json and cleanup as proper - remove obsolete site.ad file from TW and schedulers #8720
• check if input_dataset_duplicate_lumis.json is ever useful.
  • It only used in crab report to tell user that there is a non-empty list of duplicates and point to it. Could probably avoid writing when it is empty (content={}) but current way allows to distinguish no-duplicates from error-retrieving-from-WEB_DIR

[1] see also

CRABServer/src/python/ServerUtilities.py

Lines 427 to 434 in e8149dd

	def parseJobAd(filename):
	""" Parse the jobad file provided as argument and return a dict representing it
	SB: why do we have this ? the classAd object returned by classad.parse has
	the semantic of a dictionary ! Currently it is only used in cmscp.py
	and in job wrapper we are not sure to have HTCondor available.
	Note that we also have a parseAd() method inside CMSRunAnalysis.py which should
	do finely also in cmscp.py
	"""

[belforte]

How files make their way to scheduler's SPOOL_DIR and WN

thiis is what DagmanCreator has

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Lines 1233 to 1235 in e8149dd

	inputFiles = ['gWMS-CMSRunAnalysis.sh', 'submit_env.sh', 'CMSRunAnalysis.sh', 'cmscp.py', 'cmscp.sh', 'RunJobs.dag', 'Job.submit', 'dag_bootstrap.sh',
	'AdjustSites.py', 'site.ad', 'site.ad.json', 'datadiscovery.pkl', 'taskinformation.pkl', 'taskworkerconfig.pkl',
	'run_and_lumis.tar.gz', 'input_files.tar.gz']

but also

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Lines 692 to 694 in e8149dd

	Also, this prepareLocal method prepare a single "InputFiles.tar.gz" file with all the inputs files moved
	from the TW to the schedd.
	This is used by the client preparelocal command.

i.e.

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Lines 726 to 731 in e8149dd

	tf = tarfile.open('InputFiles.tar.gz', mode='w:gz')
	try:
	for ifname in inputFiles + subdags + ['input_args.json']:
	tf.add(ifname)
	finally:
	tf.close()

So the code to create InputFiles.tar.gz was possibly created for preparelocal, but eventually
DagmanSubmitter puts this in the DAGJob.jdl

transfer_input_files = InputFiles.tar.gz, subdag.jdl

The way things work is that DagmanCreator returns

CRABServer/src/python/TaskWorker/Actions/DagmanCreator.py

Line 1255 in e8149dd

return info, params, ["InputFiles.tar.gz"], splitterResult

which gets passed as input to DagmanSubmitter an in there

CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

Line 261 in e8149dd

inputFiles = args[0][2]

followed by

CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

Line 399 in e8149dd

info['inputFilesString'] = ", ".join(inputFiles + ['subdag.jdl'])

and eventually

CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

Line 540 in e8149dd

jobJDL["transfer_input_files"] = str(info['inputFilesString'])

that jobJDL object is saved in DAGJob.jdl file in TW /data/srv/tmp/_<taskname> directory

CRABServer/src/python/TaskWorker/Actions/DagmanSubmitter.py

Lines 543 to 544 in e8149dd

	with open('DAGJob.jdl', 'w', encoding='utf-8') as fd:
	print(jobJDL, file=fd)

Later on cmscp.sh, cmscp.py make there way to the grid WN by being placed in scheduler's SPOOL_DIR and then transferred by HTCondor again to the WN via this in Job.submit

transfer_input_files = CMSRunAnalysis.sh, cmscp.py, CMSRunAnalysis.tar.gz, sandbox.tar.gz, run_and_lumis.tar.gz, input_files.tar.gz, submit_env.sh, cmscp.sh

pff...what a mess

[belforte]

a more radical solution to "move cmscp.* inside tarball" would be to add two subdirectories in CRABServer/scripts (1)

scripts/
├── dagman
├── job_wrapper
└── task_process

or maybe better (2)

scripts
├── dagman
│   └── task_process
└── job_wrapper

so that in

CRABServer/cicd/crabtaskworker_pypi/buildTWTarballs.sh

Lines 76 to 84 in e8149dd

	cp -r "${CRABSERVERDIR}/scripts"/{TweakPSet.py,CMSRunAnalysis.py,task_process} .
	cp "${CRABSERVERDIR}/src/python"/{ServerUtilities.py,RucioUtils.py,CMSGroupMapper.py,RESTInteractions.py} .
	echo "Making TaskManagerRun tarball"
	tar zcf "${RUNTIME_WORKDIR}/TaskManagerRun.tar.gz" CRAB3.zip TweakPSet.py CMSRunAnalysis.py task_process ServerUtilities.py RucioUtils.py CMSGroupMapper.py RESTInteractions.py || exit 4
	echo "Making CMSRunAnalysis tarball"
	tar zcf "${RUNTIME_WORKDIR}/CMSRunAnalysis.tar.gz" WMCore.zip TweakPSet.py CMSRunAnalysis.py ServerUtilities.py CMSGroupMapper.py RESTInteractions.py || exit 4

we only copy well named directories around, like done now for task_process

and also move expanding of job_wrapper tarball

CRABServer/scripts/CMSRunAnalysis.sh

Line 49 in e8149dd

tar xmf CMSRunAnalysis.tar.gz || exit 10042

to the top wrapper script between these lines

CRABServer/scripts/gWMS-CMSRunAnalysis.sh

Lines 225 to 227 in e8149dd

	echo "======== PROXY INFORMATION FINISH at $(TZ=GMT date) ========"
	echo "======== CMSRunAnalysis.sh at $(TZ=GMT date) STARTING ========"

@novicecpp I will very much like your opinion

[belforte]

I will close this when #8718 and #8721 are merged.
Will leave further refectoring as per comment above to new issue #8727 and document "tarball movements" as per #8699 (comment) to after I have finalized #7461 and #6544 . Now tracked in #8728

[belforte]

conditions for closing met :-)