Unexpected error: list index out of range

[kinow]

From two app workflows in DestinE (cc @franra9):

2024-12-18 17:07:33,587 Calculating possible ready jobs for LUMI
2024-12-18 17:07:33,667 Traceback (most recent call last):
  File "/appl/AS/4.1.12-dev-e8877c9/bin/autosubmit", line 36, in main
    return_value = Autosubmit.parse_args()
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmit/autosubmit.py", line 711, in parse
_args
    return Autosubmit.run_experiment(args.expid, args.notransitive,args.start_time,args.start_after, args.run_o
nly_members, args.profile)
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmit/autosubmit.py", line 2240, in run_experiment
    Autosubmit.submit_ready_jobs(as_conf, job_list, platforms_to_test, packages_persistence, hold=False)
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmit/autosubmit.py", line 2513, in submit_ready_jobs
    packages_to_submit = packager.build_packages()
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmit/job/job_packager.py", line 554, in build_packages
    job.update_parameters(self._as_config,
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmit/job/job.py", line 2068, in update_parameters
    parameters = as_conf.substitute_dynamic_variables(parameters,80)
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmitconfigparser/config/configcommon.py", line 908, in substitute_dynamic_variables
    self.clean_dynamic_variables(pattern, in_the_end)
  File "/appl/AS/4.1.12-dev-e8877c9/lib64/python3.9/site-packages/autosubmitconfigparser/config/configcommon.py", line 854, in clean_dynamic_variables
    if isinstance(dynamic_var[1], list):
IndexError: list index out of range

[CRITICAL] 2024-12-18 17:07:33,667 Unexpected error: list index out of range.
 Please report it to Autosubmit Developers through Git
2024-12-18 17:07:33,668 More info at https://autosubmit.readthedocs.io/en/master/troubleshooting/error-codes.html
2024-12-18 17:07:34,018 lumi(log_recovery): Exiting.
2024-12-18 17:07:34,522 lumi-login(log_recovery): Exiting.
2024-12-18 17:07:34,847 local(log_recovery): Exiting.

[kinow]

Debugging with PDB. Logging the progress in the weekly 51 testing suite issue (to avoid information about experiments leaking here). Will update this issue later.

[kinow]

I probably won't have time to work on this today as I have to leave earlier. @isimo00 if you get bored with the mail notification PR (that had a git conflict when I checked some minutes ago 😬 ), you can have a look at this one. I only know it happens with application workflows, and when I debugged with PDB the configuration key that had '%CURRENT_OPA_CUSTOM_DIRECTIVES%'... but it's a bit complicated to reproduce it, the experiment is t01v in the ClimateDT VM.

Maybe this is something we can look together on Friday, and I would show you how given an expid and an error, one can go and find more information to reproduce the error locally and debug with IDE. 👋

[kinow]

After some tests, managed to create a local experiment with similar configuration, that fails with the same error (I used local as platform, local project, and replaced the scripts).

Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 36, in main
    return_value = Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 733, in parse_args
    return Autosubmit.inspect(args.expid, args.list, args.filter_chunks, args.filter_status,
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 1654, in inspect
    Autosubmit.generate_scripts_andor_wrappers(
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 1740, in generate_scripts_andor_wrappers
    Autosubmit.submit_ready_jobs(as_conf, job_list, platforms_to_test, packages_persistence, True,
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 2513, in submit_ready_jobs
    packages_to_submit = packager.build_packages()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/job/job_packager.py", line 554, in build_packages
    job.update_parameters(self._as_config,
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/job/job.py", line 2074, in update_parameters
    parameters = as_conf.substitute_dynamic_variables(parameters,80)
  File "/home/kinow/Development/python/micromamba/envs/autosubmit4/lib/python3.9/site-packages/autosubmitconfigparser/config/configcommon.py", line 908, in substitute_dynamic_variables
    self.clean_dynamic_variables(pattern, in_the_end)
  File "/home/kinow/Development/python/micromamba/envs/autosubmit4/lib/python3.9/site-packages/autosubmitconfigparser/config/configcommon.py", line 854, in clean_dynamic_variables
    if isinstance(dynamic_var[1], list):
IndexError: list index out of range

 [CRITICAL] Unexpected error: list index out of range.
 Please report it to Autosubmit Developers through Git
More info at https://autosubmit.readthedocs.io/en/master/troubleshooting/error-codes.html

[kinow]

Started deleting platforms not used, then jobs one by one. Then deleting the OPA_* jobs made the error go away (cc @franra9). Left just one OPA_ job, and it still gave me the error.

Then removed this line CUSTOM_DIRECTIVES: '%CURRENT_OPA_CUSTOM_DIRECTIVES%', and everything worked.

[kinow]

Minimal config to reproduce the error:

DEFAULT:
  EXPID: a030
  HPCARCH: local
  CUSTOM_CONFIG: //home/kinow/autosubmit/a030/proj/local_project/conf/bootstrap/
CONFIG:
  AUTOSUBMIT_VERSION: 4.1.12
  TOTALJOBS: 20
  MAXWAITINGJOBS: 20
PLATFORMS:
  DUMMY:
    HOST: bla
    USER: bla
JOBS:
  OPA_ENERGY_ONSHORE_1:
    CHECK: on_submission
    #CUSTOM_DIRECTIVES: '%CURRENT_OPA_CUSTOM_DIRECTIVES%'
    EXCLUSIVE: '%CURRENT_OPA_EXCLUSIVE%'
    FILE: ok.sh
    PARTITION: '%CURRENT_APP_PARTITION%'
    PLATFORM: local
    PROCESSORS: '%CURRENT_OPA_PROCESSORS%'
    RUNNING: chunk
    NAME: energy_onshore_1
    SPLITS: auto
    ADDITIONAL_FILES: []
EXPERIMENT:
  DATELIST: 19900101
  MEMBERS: fc0
  CHUNKSIZEUNIT: month
  CHUNKSIZE: 1
  NUMCHUNKS: 1
  CALENDAR: standard
  SPLITSIZEUNIT: day
PROJECT:
  PROJECT_TYPE: local
  PROJECT_DESTINATION: local_project
LOCAL:
  PROJECT_PATH: proj/local_project

[kinow]

Found a fix that works for me, or at least doesn't crash the experiment, and I can use the variable in a template (so it looks like it worked). I just need to review the PR to make sure I understand the impact that change may have, add some docs and tests, and release it and update in Autosubmit -- BSC-ES/autosubmit-config-parser#59

[mcastril]

Hello

What's exactly causing the problem? (I mean, what's not really working or what feature cannot be used?)

I see the traceback:
[CRITICAL] 2024-12-18 17:07:33,667 Unexpected error: list index out of range.

And there is a list in the configuration, but I don't think I am in the right direction

I think that we must find a way to explain users (and others as me) what's not going all well when an issue is created

[kinow]

Hello

What's exactly causing the problem? (I mean, what's not really working or what feature cannot be used?)

I see the traceback: [CRITICAL] 2024-12-18 17:07:33,667 Unexpected error: list index out of range.

And there is a list in the configuration, but I don't think I am in the right direction

I think that we must find a way to explain users (and others as me) what's not going all well when an issue is created

This is currently broken:

# commented in the example above https://github.com/BSC-ES/autosubmit/issues/2043#issuecomment-2563696755
CUSTOM_DIRECTIVES: '%CURRENT_OPA_CUSTOM_DIRECTIVES%'

Because Autosubmit treats the custom directive as a list, it enters a part of the code that raises the exception. So anyone using a placeholder/dynamic variable for CUSTOM_DIRECTIVES with this version of the AS Config Parser + AS should see this error (probably in some earlier releases too).

[kinow]

Or even better, we could say that using placeholders in CUSTOM_DIRECTIVES will result in this error, and that this can happen with other variables that are lists in AS (I believe those variables would trigger the same error)