|
| 1 | +# Computing Access |
| 2 | +_courtesy of Greg Andrews_ |
| 3 | + |
| 4 | +## iTerm2 (Mac) |
| 5 | +If you are on Mac, I prefer to user [iTerm2](https://iterm2.com/) as opposed to the stock terminal, it supports multiple tabs, is much more customizable, and has many additional features. |
| 6 | + |
| 7 | +Make sure to change your key bindings to natural language processing. |
| 8 | + |
| 9 | +## VPN Setup |
| 10 | +Refer to instructions [here](https://umassmed.sharepoint.com/sites/information-technology/SitePages/VPN-Connect.aspx) (note: you will need to login to you umassmed.edu microsoft account). |
| 11 | + |
| 12 | +## WSL (Windows) |
| 13 | +If on windows, you will have to install [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) to launch a Linux VM from which you can SSH into the ZLab servers. |
| 14 | + |
| 15 | +[WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands) |
| 16 | + |
| 17 | +From your powershell start your linux VM with: |
| 18 | +``` |
| 19 | +wsl.exe -d ubuntu |
| 20 | +``` |
| 21 | +## Update ssh config file |
| 22 | +Your ssh config file should be placed in `~/.ssh/config`. |
| 23 | + |
| 24 | +If `~/.ssh` directory does not exists, run `mkdir ~/.ssh`. |
| 25 | + |
| 26 | +On a Mac, you can create a blank ssh config file with `touch ~/.ssh/config`, and then open it in TextEdit with `Open -a TextEdit ~/.ssh/config`, then you can just copy and paste the contents below, save and exit. |
| 27 | + |
| 28 | +Replace `USERNAME` with your ZLab username. |
| 29 | + |
| 30 | +You will have to make `~/.ssh/sockets` if it does not exist with `mkdir ~/.ssh/sockets`. |
| 31 | +``` |
| 32 | +Host * |
| 33 | + TCPKeepAlive = yes |
| 34 | + ServerAliveCountMax = 3 |
| 35 | + ServerAliveInterval = 30 |
| 36 | + ForwardX11 = yes |
| 37 | + ForwardX11Trusted = yes |
| 38 | + ControlMaster auto |
| 39 | + ControlPath ~/.ssh/sockets/%r@%h:%p |
| 40 | + ControlPersist 60 |
| 41 | +
|
| 42 | +Host z010 z011 z012 z013 z014 |
| 43 | + ProxyCommand=ssh -W %h:%p -l %r bastion.wenglab.org |
| 44 | + ForwardX11 yes |
| 45 | + ForwardAgent yes |
| 46 | + ForwardX11Timeout 7d |
| 47 | + ServerAliveCountMax 3 |
| 48 | + ServerAliveInterval 15 |
| 49 | + GSSAPIAuthentication yes |
| 50 | + User USERNAME |
| 51 | +``` |
| 52 | +## Logging in |
| 53 | + |
| 54 | + |
| 55 | +If it's your first time, you will be prompted to scan the QR code with any two-factor authentication app. You should also be prompted to change your password. |
| 56 | +*** |
| 57 | +🔴 **IMPORTANT** 🔴 Save this QR code somewhere! |
| 58 | +*** |
| 59 | +I personally recommend using Microsoft Authenticator, since this is the 2FA app used by your UMass Microsoft account. However, it may also be convenient to choose a 2FA app that has a desktop client. |
| 60 | +*** |
| 61 | +🔴 **IMPORTANT** 🔴 From this point forward, every time you login, you will provide your password + two-factor code with no spaces. |
| 62 | +*** |
| 63 | +From bastion, you can then `ssh` into any of the ZLab servers: |
| 64 | +``` |
| 65 | +ssh HOSTNAME |
| 66 | +``` |
| 67 | +Replace `HOSTNAME` with one of the hostnames defined in your ssh config file (i.e. z011, z012 ... z014). |
| 68 | + |
| 69 | +# Installing Software |
| 70 | +## Install conda |
| 71 | +Conda is a great way to quickly install software and create separate environments for projects requiring different, and potentially conflicting, pieces of software. The conda (Miniforge3) installation instructions have been adapted from default installation instructions so it is available across all our machines. |
| 72 | + |
| 73 | +``` |
| 74 | +curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" |
| 75 | +bash Miniforge3-$(uname)-$(uname -m).sh -p /zata/zippy/$(whoami)/Miniforge3 |
| 76 | +source ~/.bashrc |
| 77 | +``` |
| 78 | +`$(whoami)` is evaluating to your Zlab username. |
| 79 | + |
| 80 | +While this is optional, you may want to use the `mamba` dependency resolver which tends to be faster than native `conda`. You can still use conda if you are prefer, but in the following examples will use `mamba`. You may need to run `mamba init` if this is your first time using `mamba` and then update your bashrc file with `source ~/.bashrc`. |
| 81 | + |
| 82 | +Create your first conda environment: |
| 83 | +``` |
| 84 | +mamba create -n myenv jupyterlab numpy pandas matplotlib bedtools |
| 85 | +``` |
| 86 | +## Singularity |
| 87 | +### Why _containerizing_ your environment is best practice |
| 88 | +Containers are useful in bioinformatics because they ensure reproducibility by packaging all software, dependencies, and configurations into a single unit that will run identically across different computers and systems. Hence, they solve the notorious "dependency hell" problem, allowing you to run multiple versions of tools without conflicts and easily share pipelines with others. |
| 89 | + |
| 90 | +### Build singularity image |
| 91 | +The remote docker image `clarity001/bioinformatics` contains a full suite of bioinformatics software that you will most likely need. |
| 92 | + |
| 93 | +Create the destination folder and build the singularity sandbox container: |
| 94 | +``` |
| 95 | +mkdir -p /zata/zippy/$(whoami)/bin/ |
| 96 | +singularity build --sandbox /zata/zippy/$(whoami)/bin/bioinformatics docker://clarity001/bioinformatics |
| 97 | +``` |
| 98 | +To start an interactive shell in the *writable* container (optional): |
| 99 | +``` |
| 100 | +singularity shell --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics |
| 101 | +``` |
| 102 | +# JupyterLab |
| 103 | +In the following examples: |
| 104 | +* Replace `USERNAME` with your ZLab username. |
| 105 | +* Replace `GROUP` with `zusers` if you are in ZLab, `musers` if you are in Moore Lab, or `rusers` if you are a rotation student in either group. If you are still unsure, run `groups` in your terminal. This will return a list of user groups that you belong to. Among them should be one of the previously mentioned groups. |
| 106 | +* Replace `HOSTNAME` with one of the hostnames defined in your ssh config file (i.e. z011, z012 ... z014) |
| 107 | + |
| 108 | +## Start a JupyterLab server using singularity |
| 109 | +Using the sandbox container you just built, start the JupyterLab server with the following command: |
| 110 | +``` |
| 111 | +singularity exec --writable -B /data,/zata /zata/zippy/$(whoami)/bin/bioinformatics jupyter lab --port=8888 --ip=HOSTNAME --no-browser --notebook-dir=/data/GROUP/$(whoami) |
| 112 | +``` |
| 113 | +## Start a JupyterLab server using docker |
| 114 | +First see [rootless docker setup](rootless_docker_setup) |
| 115 | +``` |
| 116 | +docker container run -it --rm -p 8888:8888 --mount type=bind,src=/data,target=/data --mount type=bind,src=/zata,target=/zata clarity001/bioinformatics jupyter-lab --port=8888 --ip=* --no-browser --allow-root --notebook-dir=/data/GROUP/$(whoami)/ |
| 117 | +``` |
| 118 | +## Start a JupyterLab server in a conda environment |
| 119 | +Activate your environment from the previous example: |
| 120 | +``` |
| 121 | +conda activate myenv |
| 122 | +``` |
| 123 | +Start your jupyterlab server: |
| 124 | +``` |
| 125 | +jupyter-lab --port=8888 --ip=HOSTNAME --no-browser |
| 126 | +``` |
| 127 | +## Accessing your JupyterLab server |
| 128 | +Access the notebook server from another terminal on your computer with `ssh -N -L 8888:HOSTNAME:8888 USERNAME@HOSTNAME`. |
| 129 | + |
| 130 | +If you are curious, this is what the previous command is doing: |
| 131 | +``` |
| 132 | +ssh # The secure shell program that creates encrypted connections |
| 133 | +-N # Flag that means "don't execute a remote command/shell" - just forward ports |
| 134 | +-L # Flag for "local port forwarding" |
| 135 | +8888: # The local port on your computer |
| 136 | +HOSTNAME: # The destination server's address (check your ssh config for available hosts) |
| 137 | +8888 # The remote port on the destination server |
| 138 | +USERNAME@HOSTNAME # Username and server address to login to |
| 139 | +``` |
| 140 | +You can then open your notebook server in your favorite web browser by navigating to http://127.0.0.1:8888/lab. |
| 141 | + |
| 142 | +## Common issues with JupyterLab |
| 143 | +### Unable to delete files |
| 144 | +This is an issue that has come up before. To fix, you first need to find you jupyter config file. It will most likely be located at `/home/USERNAME/.jupyter`. First, change into that directory: |
| 145 | +``` |
| 146 | +cd /home/$(whoami)/.jupyter |
| 147 | +``` |
| 148 | +If you cannot find `jupyter_server_config.py` in that directory, run: |
| 149 | +``` |
| 150 | +jupyter server --generate-config |
| 151 | +``` |
| 152 | +Edit `jupyter_server_config.py` with your preferred text editor: |
| 153 | +``` |
| 154 | +vim jupyter_server_config.py |
| 155 | +``` |
| 156 | +Look for the config option `c.FileContentsManager.delete_to_trash`. If it is set to `True`, change it to `False`. Now you should be able to delete files in JupyterLab |
| 157 | +(rootless_docker_setup)= |
| 158 | +# Rootless Docker setup |
| 159 | +## Why use rootless Docker? |
| 160 | +Rootless Docker enables users to run containers without administrator privileges, making it ideal for shared HPC environments where granting root access to multiple users would pose security risks. |
| 161 | +## Setup |
| 162 | +``` |
| 163 | +mkdir -p ~/.config/docker/ |
| 164 | +echo '{"data-root":"/rootless/docker/'$(whoami)'/docker"}' > ~/.config/docker/daemon.json |
| 165 | +dockerd-rootless-setuptool.sh install |
| 166 | +``` |
0 commit comments