Notebook Server Recipes fail in Disconnected Environment #385

jmturco17 · 2024-05-27T19:01:06Z

I'm attempting to deploy ArcGIS Enterprise with an ArcGIS Server Notebook role in a air-gapped (disconnected) environment. The Notebook server chef-solo run appears to be reaching out to "index.rubygems.org" which fails. Details on the issues can be found below:

FATAL: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17'
-------Begin output of ['bundle', 'install'] ------
STDOUT:
STDERR: Could not reach host index.rubgems.org. Check your network connection and try again
-------End output of ["bundle", "install"]
Ran ["bundle", "install"] returned 17

jmturco17 · 2024-05-29T17:57:04Z

CInc Version is 17.0.0

cameronkroeker · 2024-05-29T18:20:53Z

CInc Version is 17.0.0

Thanks @jmturco17. Can you also give Cinc client 18 a try? Curious to see if the issue is reproducible with the latest version of Cinc client.

cameronkroeker · 2024-05-31T17:37:33Z

I suspect the issue could be related to the third party docker cookbook that the notebook server cookbook uses to install docker engine. This looks like it could be related:

sous-chefs/docker#682

Additionally, there is this requirement of a network accessible web server containing the docker binaries:

https://github.com/sous-chefs/docker/blob/main/README.md#requirements

Might need to install/configure docker using a different method and set the following attribute in the json config file to false:

node['arcgis']['notebook_server']['install_docker'] = If set to true, the arcgis-notebooks::docker recipe installs the Docker engine. Default value is false for RHEL Linux and true otherwise.

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#attributes

jmturco17 · 2024-05-31T18:35:35Z

I'm going to be trying out CINC 18.4.12 next week to give it a try.

I'm manually installing docker before even getting to the Notebook Server installation. I've also removed the docker install fro the run-list and the issues persists.

cameronkroeker · 2024-05-31T19:36:47Z

I'm going to be trying out CINC 18.4.12 next week to give it a try.

I'm manually installing docker before even getting to the Notebook Server installation. I've also removed the docker install fro the run-list and the issues persists.

Please share your recipe run-list so we can get a better picture of what all is being ran. If possible your json config file as well with sensitive information such as passwords/machine names removed.

jmturco17 · 2024-06-04T00:03:12Z

Here is the whole JSON file minues the sensitive data

{
    "arcgis": {
        "version": "<version>",
        "run_as_user": "arcgis",
        "repository": {
            "archives": "/opt/software/archives",
            "setups": "/opt/software/setups"
        },
        "notebook_server": {
            "install_dir": "/opt",
            "install_system_requirements": true,
            "install_samples_data": true,
            "authorization_file": "/opt/software/authorization_files/<version>/notebook_advanced_authorization.eslf",
            "license_level": "standard",
            "admin_username": "<ags_username>",
            "admin_password": "<ags_password>",
            "directories_root": "/gisdata/notebookserver/directories",
            "workspace": "/gisdata/notebookserver/directories/arcgisworkspace",
            "config_store_type": "FILESYSTEM",
            "config_store_connection_string": "/gisdata/notebookserver/config-store",      
            "log_level": "WARNING",
            "log_dir": "/opt/arcgis/notebookserver/usr/logs",
            "configure_autostart": true,
            "system_properties": { }
        }
    },
    "run_list": [
        "recipe[arcgis-enterprise::system]",
        "recipe[arcgis-notebooks::docker]",
        "recipe[arcgis-notebooks::iptables]",
        "recipe[arcgis-notebooks::restart_docker]",
        "recipe[arcgis-notebooks::server]"
    ]
}

Also, please note, I've tried to slim this down be removing docker and system from the run_list and still received the issue.

jmturco17 · 2024-06-09T21:02:33Z

I can also confirm that upgrading the the latest chef recipe book DOES NOT resolve the issue.

cameronkroeker · 2024-06-10T21:42:51Z

Thanks @jmturco17 for the additional information. I dug into this a bit and discovered the following gems are trying to be installed from the internet:

As a workaround those can be downloaded on a machine that has internet access and copied over to the air-gapped environment, then installed using the following commands:

/opt/cinc/embedded/bin/gem install --local /tmp/excon-0.110.0.gem

/opt/cinc/embedded/bin/gem install --local /tmp/docker-api-2.2.0.gem

Next you will want to add skip_gem_metadata_installation to the /etc/cinc/client.rb file (or if using chef /etc/chef/client.rb file) and set it to true so that it skips installing gems from the internet:

# The client.rb file specifies how Chef Infra Client is configured on a node
# See https://docs.chef.io/config_rb_client/ for detailed configuration options
#
# Minimal example configuration:
# node_name  "THIS_NODE_NAME"
# chef_server_url  "https://CHEF.MYCOMPANY.COM/organizations/MY_CHEF_ORG"
# chef_license  "accept"
file_cache_path "/var/cinc/cache"
gem_installer_bundler_options "--local"
skip_gem_metadata_installation true

For more information about the Chef client.rb: https://docs.chef.io/config_rb_client/.

cameronkroeker · 2024-06-13T15:46:35Z

Hi @jmturco17,

Here is a more detailed summary of using the following recipes in a disconnected environment.

"run_list": [
        "recipe[arcgis-enterprise::system]",
        "recipe[arcgis-notebooks::docker]",
        "recipe[arcgis-notebooks::iptables]",
        "recipe[arcgis-notebooks::restart_docker]",
        "recipe[arcgis-notebooks::server]"
]

arcgis-enterprise::system

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-enterprise#system

Installs ruby gem multipart-post from internet
- https://rubygems.org/gems/multipart-post/versions/2.4.1
- To install locally:
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/multipart-post-2.4.1.gem
- To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
  - https://docs.chef.io/config_rb_client/
Depending on OS platform it will install missing packages from the internet, ensure these are installed so that they will be skipped. The list of packages can be found here:

arcgis-cookbook/cookbooks/arcgis-enterprise/attributes/default.rb

Lines 60 to 78 in c92c776

    
           default['arcgis']['packages'] = 
        
             case node['platform'] 
        
             when 'redhat', 'centos', 'amazon', 'oracle', 'rocky' 
        
               node['arcgis']['configure_autofs'] ? 
        
                 ['gettext', 'nfs-utils', 'autofs'] : 
        
                 ['gettext'] 
        
             when 'almalinux' 
        
               node['arcgis']['configure_autofs'] ? 
        
                 ['gettext', 'glibc-langpack-en', 'nfs-utils', 'autofs'] : 
        
                 ['gettext', 'glibc-langpack-en'] 
        
             when 'suse' 
        
               node['arcgis']['configure_autofs'] ? 
        
                 ['gettext-runtime', 'autofs'] : 
        
                 ['gettext-runtime'] 
        
             else 
        
               node['arcgis']['configure_autofs'] ?  
        
                 ['gettext-base', 'libxrender1', 'libxtst6', 'libxi6', 'autofs'] : 
        
                 ['gettext-base', 'libxrender1', 'libxtst6', 'libxi6'] 
        
             end

arcgis-notebooks::docker

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#docker

Requires the following 2 ruby gems
- https://rubygems.org/gems/excon
- https://rubygems.org/gems/docker-api/versions/2.2.0
- To install locally:
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/excon-0.110.0.gem
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/docker-api-2.2.0.gem
  - To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
  - https://docs.chef.io/config_rb_client/
Downloads remote_file docker binaries (https___download_docker_com_linux_ubuntu_gpg)
- There isn't a straight forward workaround here other than perhaps hosting it localling
  - https://github.com/sous-chefs/docker/blob/main/README.md#requirements
  - I suggest setting node['arcgis']['notebook_server']['install_docker'] to false in your json config so that it skips downloading/installing docker. Docker will need to be installed/configured outside of Chef prior.

arcgis-notebooks::iptables

Requires iptables package
To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/

arcgis-notebooks::restart_docker

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#restart_docker

To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/

recipe[arcgis-notebooks::server]

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#server

To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/
Requires the setup archive installation files
- https://esri.github.io/arcgis-cookbook/templates/arcgis-notebook-server/11.3.html#required-arcgis-software-repository-content
Requires either advanced or standard notebook server license file (either .ecp or .ESLF)

When deploying in cloud environments such as AWS, it might be best or easiest to first create an image (AMI) with the gem files, linux packages, docker and chef/cinc client all installed. Then use that AMI in the disconnected environment.

Thanks,
Cameron K.

jmturco17 · 2024-06-13T16:48:01Z

Thanks Cameron, I saw your post earlier and have been working to get that running in the last day or so. I'll let you know how it all looks hopefully early next week once I'm able to test again.

cameronkroeker · 2024-06-13T17:26:50Z

@jmturco17 I should also mention, there is an issue with using a password protected ESLF with arcgis-notebooks cookbook. It's missing the authorization_options attribute which is used for passing in the password of the ESLF. Here are a couple workarounds:

Workaround 1: Simplest, but requires hardcoding password in cookbook which is not ideal from a security perspective.

Modify:

arcgis-cookbook/cookbooks/arcgis-notebooks/resources/server.rb

Line 316 in c92c776

args = "-f \"#{@new_resource.authorization_file}\""

To:
args = "-f \"#{@new_resource.authorization_file}\" -p \"yourpassword\""

Workaround 2: Recommended approach, but requires more changes than workaround 1.

Note: This is the fix that will be used in a future release of the cookbooks. The same issue is also in arcgis-video, arcgis-geoevent, arcgis-workflow-manager, and arcgis-mission cookbooks.

1. Introduce new authorization_options attribute

In:

arcgis-cookbook/cookbooks/arcgis-notebooks/resources/server.rb

Line 26 in c92c776

attribute :setup_archive, :kind_of => String

Add:

attribute :authorization_options, :kind_of => String

2. Pass in the authorization_options attribute in args

Change:

arcgis-cookbook/cookbooks/arcgis-notebooks/resources/server.rb

Line 316 in c92c776

args = "-f \"#{@new_resource.authorization_file}\""

To:

args = "-f \"#{@new_resource.authorization_file}\" #{@new_resource.authorization_options}"

3. Modify server and server_node recipes to use the attribute:

Add:

authorization_options node['arcgis']['notebook_server']['authorization_options']

For example:

arcgis_notebooks_server 'Authorize ArcGIS Notebook Server' do
  authorization_file node['arcgis']['notebook_server']['authorization_file']
  authorization_file_version node['arcgis']['notebook_server']['authorization_file_version']
  authorization_options node['arcgis']['notebook_server']['authorization_options']
  retries 2
  retry_delay 30
  notifies :stop, 'arcgis_notebooks_server[Stop ArcGIS Notebook Server]', :immediately
  action :authorize
end

4. Add and Set default value for authorization_options to empty string.

In:
https://github.com/Esri/arcgis-cookbook/blob/main/cookbooks/arcgis-notebooks/attributes/default.rb

Add:
notebook_server['authorization_options'] = ""

5. Define authorization_options attribute in your json configuration file.

"notebook_server": {
   "install_dir": "/opt",
   "install_system_requirements": true,
   "install_samples_data": true,
   "authorization_file": "/opt/software/authorization_files/<version>/notebook_advanced_authorization.eslf",
   "authorization_options": "-p theESLFPassword",
   "license_level": "standard",
   "admin_username": "<ags_username>",
   "admin_password": "<ags_password>",
   "directories_root": "/gisdata/notebookserver/directories",
   "workspace": "/gisdata/notebookserver/directories/arcgisworkspace",
   "config_store_type": "FILESYSTEM",
   "config_store_connection_string": "/gisdata/notebookserver/config-store",      
   "log_level": "WARNING",
   "log_dir": "/opt/arcgis/notebookserver/usr/logs",
   "configure_autostart": true,
   "system_properties": { }
}

jmturco17 · 2024-06-15T14:15:58Z

I made the changes in the /etc/cinc/client.rb file and cinc looks like it's still reaching out

cameronkroeker · 2024-06-17T16:21:35Z

skip_gem_metadata_installation true

You can pass it in directly via the command line as well, which overrides the /etc/cinc/client.rb. For example:

cinc-client -z -j your.json --config-option "skip_gem_metadata_installation=true"

or

cinc-solo -j your.json --config-option "skip_gem_metadata_installation=true"

jmturco17 · 2024-06-18T02:03:02Z

Thanks Cameron! The command line option seemed to take, working on the rest of the installation steps now!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Notebook Server Recipes fail in Disconnected Environment #385

Notebook Server Recipes fail in Disconnected Environment #385

jmturco17 commented May 27, 2024

jmturco17 commented May 29, 2024

cameronkroeker commented May 29, 2024

cameronkroeker commented May 31, 2024

jmturco17 commented May 31, 2024

cameronkroeker commented May 31, 2024

jmturco17 commented Jun 4, 2024 •

edited by cameronkroeker

Loading

jmturco17 commented Jun 9, 2024

cameronkroeker commented Jun 10, 2024 •

edited

Loading

cameronkroeker commented Jun 13, 2024

jmturco17 commented Jun 13, 2024

cameronkroeker commented Jun 13, 2024 •

edited

Loading

jmturco17 commented Jun 15, 2024

cameronkroeker commented Jun 17, 2024 •

edited

Loading

jmturco17 commented Jun 18, 2024

Notebook Server Recipes fail in Disconnected Environment #385

Notebook Server Recipes fail in Disconnected Environment #385

Comments

jmturco17 commented May 27, 2024

jmturco17 commented May 29, 2024

cameronkroeker commented May 29, 2024

cameronkroeker commented May 31, 2024

jmturco17 commented May 31, 2024

cameronkroeker commented May 31, 2024

jmturco17 commented Jun 4, 2024 • edited by cameronkroeker Loading

jmturco17 commented Jun 9, 2024

cameronkroeker commented Jun 10, 2024 • edited Loading

cameronkroeker commented Jun 13, 2024

arcgis-enterprise::system

arcgis-notebooks::docker

arcgis-notebooks::iptables

arcgis-notebooks::restart_docker

recipe[arcgis-notebooks::server]

jmturco17 commented Jun 13, 2024

cameronkroeker commented Jun 13, 2024 • edited Loading

Workaround 1: Simplest, but requires hardcoding password in cookbook which is not ideal from a security perspective.

Workaround 2: Recommended approach, but requires more changes than workaround 1.

1. Introduce new authorization_options attribute

2. Pass in the authorization_options attribute in args

3. Modify server and server_node recipes to use the attribute:

4. Add and Set default value for authorization_options to empty string.

5. Define authorization_options attribute in your json configuration file.

jmturco17 commented Jun 15, 2024

cameronkroeker commented Jun 17, 2024 • edited Loading

jmturco17 commented Jun 18, 2024

jmturco17 commented Jun 4, 2024 •

edited by cameronkroeker

Loading

cameronkroeker commented Jun 10, 2024 •

edited

Loading

cameronkroeker commented Jun 13, 2024 •

edited

Loading

cameronkroeker commented Jun 17, 2024 •

edited

Loading