[bugfix] Evaluate Slurm job nodelist lazily

[vkarak]

The nodelist evaluation was being triggered early by logging, when we update the log record with all the test attributes:

reframe/reframe/core/logging.py

Lines 832 to 846 in b1c8970

	for attr, alt_name in check_type.loggable_attrs():
	extra_name = alt_name or attr
	with suppress_deprecations():
	# In case of AttributeError, i.e., the variable is undefined,
	# we set the value to None
	val = getattr(self.check, attr, None)
	if attr in check_type.raw_params:
	# Attribute is parameter, so format it
	val = check_type.raw_params[attr].format(val)
	key = f'check_{extra_name}'
	self.extra['__rfm_loggable_attrs__'].append(key)
	self.extra[key] = val

The problem is that in some cases Slurm does not populate the nodelist of the job immediately, so we might end with an incomplete node list in the test job. We fix this by evaluating lazily the job's nodelist after the job has finished.

Fixes #3018.

@stefaniereuter could you please check if this fixes your problem?

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (3e3f4c5) 86.71% compared to head (8f22223) 86.71%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3022   +/-   ##
=======================================
  Coverage   86.71%   86.71%           
=======================================
  Files          61       61           
  Lines       12002    12002           
=======================================
  Hits        10407    10407           
  Misses       1595     1595           

Files	Coverage Δ
reframe/core/schedulers/slurm.py	52.92% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stefaniereuter]

Thank you so much @vkarak. I will try this as soon as our system is back from maintenance.

[vkarak]

@stefaniereuter did you manage to try this out?

[stefaniereuter]

I am so sorry I haven't had a chance to test yet as our system will still be down for probably another week. 26 Oct 2023 14:49:16 Vasileios Karakasis ***@***.***>:

…

@stefaniereuter[https://github.com/stefaniereuter] did you manage to try this out? — Reply to this email directly, view it on GitHub[#3022 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AXOHEN4EFAFGECZSKKOVV73YBJL4XAVCNFSM6AAAAAA6H3HMQCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRGA2TONJUHE]. You are receiving this because you were mentioned. [Tracking image][https://github.com/notifications/beacon/AXOHEN5FTVADCSMKNLZQOOLYBJL4XA5CNFSM6AAAAAA6H3HMQCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTKFDEA2.gif]

[vkarak]

@stefaniereuter Did you have a chance to test this? We will merge this PR so as to make it to the 4.4.1 release. If the problem is not fixed, feel free to reopen the issue.

[stefaniereuter]

sorry for the very late response. Yes we were finally able to test this last week and it seem to fix our issue, thank you so much