Skip to content

Setup instructions

Jump to bottom
Gaudeval edited this page Jul 22, 2022 · 18 revisions

Please note that this project is an active work in progress, documentation and features are subject to frequent changes.

The following guide was produced using the Ubuntu 22.04 Server Linux distribution for the cluster setup. Please adapt command and configuration parameters to your own distribution. The Controller and Worker should use the same version of Slurm to ensure they are compatible (this is easily achieved by using the same Linux distribution across all cluster members).

TODO: Lay down some basic notations: configuration parameters, commands to run, configuration file contents...

Cluster roles

The pakupi setup guide refers to a number of roles assumed by machines in the cluster. We recommend the use of distinct machines for each role, to mitigate the impact of reboots or issues during installation. This guide identifies the following roles:

  • Host: the machine used to configure the cluster
  • Controller: the main node for providing cluster services and interracting with the cluster
  • Worker: any node whose processsing power is made available to cluster

Preparing the Host

The Host is the main entry point to the cluster. It will ensure all machines in the cluster are correctly configured through Ansible. The Host should have access to the Internet to download the required packages, and network access to the cluster.

  • Install Ansible: Ansible is an IT automation framework which allows users to manage an inventory of machines, configure systems, and deploy software. The pakupi setup process relies on Ansible to check the configuration of the cluster, and run additional scripts if required. For more information on how to setup Ansible, see the Ansible installation guide. Under Ubuntu 22.04 you can install Ansible using the command: apt install ansible.

    CHECK: Run the following command in your terminal to ensure Ansible is correctly installed: ansible --version

  • Install pakupi Ansible requirements: The pakupi setup process uses community-provided Ansible roles and collections (see the Ansible user guide on roles). The requirements are described in the requirements-galaxy.yml at the root of the repository. The packages are available for download and review through Ansible galaxy. You can install the required packages using the command: ansible-galaxy install -r requirements-galaxy.yml.

    CHECK: Running the installation command: ansible-galaxy install -r requirements-galaxy.yml should report Nothing to do

  • Prepare a SSH key: SSH allows remote connections to the machines in the cluster, and it is used by Ansible to configure the system. Authentication through SSH can be password-based, using the target account password, or key-based, using a list of authorised public keys. Key-based authentication is recommended as it does not prompt the user for a password. To generate a new SSH key, use the ssh-keygen command. The created key will be placed in your home directory under ~/.ssh/id_rsa.pub and id_rsa for the private key (which should not be distributed).

    If you reuse an existing key, or specify a different target files for the generated key using the -f flag of ssh-keygen, please remember to add the key to your SSH agent (see man ssh-add).

    CHECK: Your public and private key files exist (respectively under ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa.pub by default).

Preparing the Controller

The Controller is the main node for interacting with the cluster. It schedules work on the workers, and provide required service for cluster operation such as file sharing, job submission, dashboard, etc. During its configuration, the Controller should have access to the Internet to download the required packages, and network access to the cluster.

Setup the Operating System

The pakupi setup scripts assume the Controller is running the Ubuntu 22.04 Server Linux distribution. The scripts may be adapted to your favourite flavour of Linux, but please make sure the same version of Slurm will be available on the Controller and the Workers. There are no minimum requirements on the Controller, but it should have enough computing power, memory, and storage to provide the required services. Please consult the Ubuntu Server Installation Guide for more information. Note that a Raspberry Pi could be devoted to that role, preferrably with external storage. We discuss the setup process in the Worker section of this guide.

The following need to be considered during the setup:

  • Configure a fixed IP for the Controller on the interface used to interact with the cluster. The Controller needs a fixed address to generate the workers configuration. It will provide dynamic IP allocation for other cluster machines.
  • Create a user with administrative rights. This should be the default for the user created during installation. The pakupi setup will need to install new packages on the Controller and configure services which required administrative rights. Please make note of the created username and password as those will be required to run Ansible commands.
  • Enable SSH access on the Controller. SSH allows remote access to the Controller for configuration and monitoring purposes.

TODO: Check if enabled by default TODO: Check if can be enabled on Pi using file on /boot partition TODO: Check if can be enabled on Pi by adding packages to be installed on /boot partition

  • Add Host SSH Key to authorised keys to allow password-less access from the Host to the Controller. (during setup or using ssh-copy-id)

TODO: Check if can be set on Pi by adding key on /boot partition

CHECK: You should be able to connect to the Controller through ssh. ssh <controller-user>@<controller-ip>. Might prompt for recognising controller on first connection.

Configuring a network connection

/boot/firmware/network-config or /etc/netplan/...

Adding the Controller to the inventory

Illustrate yaml and ini format

Show relevant variables if required

  • ansible user
  • ansible host

Under group/name mocha-worker

Setup DHCP server

Run 01-dhcp

Specify -K flag optionally

Identify commands to check the DHCP server is running

Preparing a worker

Setup the Operating System

- Keep user/password
- Setup worker for SSH access 
- Setup DHCP on Worker (`/boot/firmware/network-config`)

Add SSH key

Configuring the network connection

Adding a worker to the inventory

Add SSH key if not during OS setup

Configure the cluster

  • Run NIS setup playbook (02)
  • Run slurm setup playbook (03)
  • Run NFS setup playbook (04)
  • Run worker configuration playbook (05)

Configure the dashboard

Clone this wiki locally