Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processing optional schema parameters #56

Open
maany opened this issue Oct 25, 2019 · 1 comment
Open

processing optional schema parameters #56

maany opened this issue Oct 25, 2019 · 1 comment
Assignees
@maany
Labels
bug Something isn't working

Comments

@maany
Copy link
Member

maany commented Oct 25, 2019

Describe the bug
A clear and concise description of what the bug is.
What input files were supplied to the compiler?
What output/error did you get?
Error:

KeyError: 'supplemental_config'

Input

### url of schema
schema:  "https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml"

### Variable declaration:
global_variables:
  - &lightweight_component01_ip_address 188.185.112.251
  - &lightweight_component01_fqdn simple-condor-ce.cern.ch
  - &lightweight_component02_ip_address 188.185.84.16
  - &lightweight_component02_fqdn simple-condor-batch.cern.ch
  - &lightweight_component03_ip_address 188.185.70.127
  - &lightweight_component03_fqdn simple-condor-worker.cern.ch
  - &lightweight_component04_ip_address 188.185.85.178
  - &lightweight_component04_fqdn simple-condor-submit.cern.ch
htcondor_ce_runtime_variables:
  - &htcondor_ce_runtime_var_ce_host simple-condor-ce.cern.ch

htcondor_batch__runtime_variables:
  - &htcondor_runtime_var_batch_host simple-condor-batch.cern.ch

site:
  name: 'simple-dev-site'
  email: '[email protected]'
  latitude: 46.3
  longitude: 6.2
  location: CERN, Geneva, Switzerland
  description: 'A demo site demonstrating the SIMPLE framework for site setup'
  website: 'https://home.cern'
  support_website: 'https://groups.google.com/forum/#!forum/wlcg-lightweight-sites'
  support_email: '[email protected]'
  security_email: '[email protected]'
  grid: 'wlcg' #site_other_grid: str(), wlcg, egi,osg
  tier: 3
  bdii_host: bdii.cern.ch
  cvmfs_http_proxy_list:
    - "http://cvmfs.cat.cbpf.br:3128"
    - "http://cvmfs-sec.cat.cbpf.br:3128"
  use_argus: no

preferred_tech_stack:
  level_1_configuration: puppet
  level_2_configuration: sh
  container_orchestration: docker-swarm
  container: docker

site_infrastructure:
  - fqdn: *lightweight_component01_fqdn
    ip_address: *lightweight_component01_ip_address
  - fqdn: *lightweight_component02_fqdn
    ip_address: *lightweight_component02_ip_address
  - fqdn: *lightweight_component03_fqdn
    ip_address: *lightweight_component03_ip_address
  - fqdn: *lightweight_component04_fqdn
    ip_address: *lightweight_component04_ip_address

lightweight_components:
  - type: compute_element
    name: HTCondor-CE
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce"
    repository_revision: "master"
    execution_id: 0
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    deploy:
      - node: *lightweight_component01_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    preferred_tech_stack:
      level_2_configuration: ""

  - name: HTCondor-Batch
    type: batch_system
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch"
    repository_revision: "master"
    execution_id: 1
    deploy:
      - node: *lightweight_component02_fqdn
        container_count: 1
    config:
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""

  - type: worker_node
    name: HTCondor-Execute
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker"
    repository_revision: "master"
    execution_id: 2
    deploy:
      - node: *lightweight_component03_fqdn
        container_count: 1
      - node: *lightweight_component04_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""

supported_virtual_organizations:
  - *default_vo_alice
  - *default_vo_dteam
  - *default_vo_ops

voms_config:
  - voms_fqan: '/alice'
    pool_accounts:
      - *default_pool_accounts_alice
    vo: *default_vo_alice
  - voms_fqan: '/alice/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_alice
      - *default_pool_accounts_alicesgm
    vo: *default_vo_alice
    comment: sgm
  - voms_fqan: '/dteam'
    pool_accounts:
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
  - voms_fqan: '/dteam/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_dteamsgm
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
    comment: sgm
  - voms_fqan: '/ops'
    vo: *default_vo_ops
    pool_accounts:
      - *default_pool_accounts_ops
  - voms_fqan: '/ops/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_opssgm
      - *default_pool_accounts_ops
    vo: *default_vo_ops
    comment: sgm

Output

/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/bin/python /Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py ./tests/resources/condor-config.yaml -o ./tests/output.yaml -s ./tests/schema.yaml
[{'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker', 'revision': 'master'}]
RUNTIME$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Param 1: schema
     - required: True
     - use_default: True
     - specified: True
     - default_value: http://github.com/WLCG-lightweight-Sites/simple_grid_site_repo/simple-base-schema.yaml
     - user_defined_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
     - final_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
Param 2: global_variables
     - required: False
     - use_default: False
     - specified: True
     - user_defined_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
     - final_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
Param 3: site_infrastructure
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-ce.cern.ch
          - final_value: simple-condor-ce.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.112.251
          - final_value: 188.185.112.251
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-batch.cern.ch
          - final_value: simple-condor-batch.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.84.16
          - final_value: 188.185.84.16
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-worker.cern.ch
          - final_value: simple-condor-worker.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.70.127
          - final_value: 188.185.70.127
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-submit.cern.ch
          - final_value: simple-condor-submit.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.85.178
          - final_value: 188.185.85.178
Param 4: lightweight_components
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-CE
          - final_value: HTCondor-CE
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: compute_element
          - final_value: compute_element
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 0
          - final_value: 0
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-ce.cern.ch
               - final_value: simple-condor-ce.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
Traceback (most recent call last):
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 185, in <module>
    main()
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 181, in main
    execute_compiler(site_level_configuration_file, output, schema)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 163, in execute_compiler
    phase_5_output_file = phase_5(phase_4_output, runtime_vars, yaml)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 63, in phase_5
    meta_info_file = base_repo_file("meta_info")
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 199, in process_complete_schema
    return get_final_config(config, expected_params, defaults, meta_info, expandable_types)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 132, in get_final_config
    indent = indent + 1)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 106, in get_final_config
    config = config[param],
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/lib/python2.7/site-packages/ruamel/yaml/comments.py", line 757, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'supplemental_config'

Process finished with exit code 1

https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/blob/master/simple_grid_yaml_compiler/compiler/processor_config_schemas.py#L105
If param is optional in this line (supplemental_config, lifecycle_hooks, preferred_tech_stack), then KeyError will be thrown.

To Reproduce
Run against master branch of simple_grid_site_repo

BASE_REPO_URL = "https://github.com/WLCG-Lightweight-Sites/simple_grid_site_repo"
BASE_REPO_REVISION = "master"

Expected behavior

Desktop (please complete the following information):

  • OS: MacOS
  • Python 2.7
  • Compiler: 1.0.0 and master

Additional context
Add any other context about the problem here.
@maany maany added the bug Something isn't working label Oct 25, 2019
@maany maany self-assigned this Oct 25, 2019
@maany
Copy link
Member Author

maany commented Oct 25, 2019

More Info:

        if param_type in expandable_types:
            config[param] = get_final_config(
                    config = config[param],
                    expected_params = expandable_types[param_type],
                    defaults = defaults,
                    meta_info = meta_info,
                    expandable_types = expandable_types,
                    indent = indent + 1)

            continue

        else:
            match = re.match("list\((.*)\)", param_type)

            if match and match.group(1) in expandable_types:
                final_values = []
                base_type    = match.group(1)

                for index, value in enumerate(config[param]):
                    if index:
                        print "     " * (indent + 1) + "-----------"

                    final_value = get_final_config(
                        config = value,
                        expected_params = expandable_types[base_type],
                        defaults = defaults,
                        meta_info = meta_info,
                        expandable_types = expandable_types,
                        indent = indent + 1)

                    final_values.append(final_value)

                config[param] = final_values
                continue
        is_required  = expected_params[param]["required"]
        use_default  = expected_params[param]["use_default"]
        is_specified = True if param in config else False

supp_config is expandable in the base_schema, but if it is not specified in the site_level_config, the value of config is None in the last line of this script.
This should be changed to

is_specified = False is config or param not in config else True

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maany maany

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@maany