Skip to content

Commit eb50d85

Browse files
casparvlcasparvl
authored
Merge pull request #157 from bedroge/private_s1
Update page about setting up a private Stratum 1
2 parents 899e4e0 + 9e97d4f commit eb50d85

File tree

2 files changed

+122
-117
lines changed

2 files changed

+122
-117
lines changed

‎docs/filesystem_layer/stratum1.md

+70-111
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,66 @@
11
# Setting up a Stratum 1
22

3-
Setting up a Stratum 1 involves the following steps:
4-
5-
- set up the Stratum 1, preferably by running the Ansible playbook that we provide;
6-
- request a Stratum 0 firewall exception for your Stratum 1 server;
7-
- request a `<your site>.stratum1.cvmfs.eessi-infra.org` DNS entry;
8-
- open a pull request to include the URL to your Stratum 1 in the EESSI configuration.
9-
10-
The last two steps can be skipped if you want to host a "private" Stratum 1 for your site.
11-
3+
The EESSI project provides a number of geographically distributed public Stratum 1 servers that you can use to make EESSI available on your machine(s).
4+
It is always recommended to have a local caching layer consisting of a few Squid proxies.
5+
If you want to be even better protected against network outages and increase the bandwidth between your cluster nodes and the Stratum 1 servers,
6+
you could also consider setting up a local (private) Stratum 1 server that replicates the EESSI CVMFS repository.
7+
This guarantees that you always have a full and up-to-date copy of the entire stack available in your local network.
128

139
## Requirements for a Stratum 1
1410

1511
The main requirements for a Stratum 1 server are a good network connection to the clients it is going to serve,
16-
and sufficient disk space. For the EESSI repository, a few hundred gigabytes should suffice, but for production
17-
environments at least 1 TB would be recommended.
12+
and sufficient disk space. As the EESSI repository is constantly growing, make sure that the disk space can easily be extended if necessary.
13+
Currently, we recommend to have at least 1 TB available.
1814

1915
In terms of cores and memory, a machine with just a few (~4) cores and 4-8 GB of memory should suffice.
2016

21-
Various Linux distributions are supported, but we recommend one based on RHEL 7 or 8.
17+
Various Linux distributions are supported, but we recommend one based on RHEL 8 or 9.
2218

23-
Finally, make sure that ports 80 (for the Apache web server) and 8000 are open.
19+
Finally, make sure that ports 80 and 8000 are open to clients.
2420

2521

26-
## Step 1: set up the Stratum 1
22+
## Configure the Stratum 1
2723

28-
The recommended way for setting up an EESSI Stratum 1 is by running the Ansible playbook `stratum1.yml`
29-
from the [filesystem-layer repository on GitHub](https://github.com/EESSI/filesystem-layer).
24+
Stratum 1 servers have to synchronize the contents of their CVMFS repositories regularly, and usually they replicate from a CVMFS Stratum 0 server.
25+
In order to ensure the stability and security of the EESSI Stratum 0 server, it has a strict firewall, and only the EESSI-maintained public Stratum 1 servers are allowed to replicate from it.
26+
However, EESSI provides a synchronisation server that can be used for setting up private Stratum 1 replica servers, and this is available at `http://aws-eu-west-s1-sync.eessi.science`.
27+
28+
!!! warn Potential issues with intrusion prevention systems
29+
In the past we have seen a few occurrences of data transfer issues when files were being pulled in by or from a Stratum 1 server.
30+
In such cases the `cvmfs_server snapshot` command, used for synchronizing the Stratum 1, may break with errors like `failed to download <URL to file>`.
31+
Trying to manually download the mentioned file with `curl` will also not work, and result in errors like:
32+
```
33+
curl: (56) Recv failure: Connection reset by peer
34+
```
35+
In all cases this was due to an intrusion prevention system scanning the associated network, and hence scanning all files going in or out of the Stratum 1.
36+
Though it was a false-positive in all cases, this breaks the synchronization procedure of your Stratum 1.
37+
If this is the case, you can try switching to HTTPS by using `https://aws-eu-west-s1-sync.eessi.science` for synchronizing your Stratum 1.
38+
Even though there is no advantage for CVMFS itself in using HTTPS (it has built-in mechanisms for ensuring the integrity of the data),
39+
this will prevent the described issues, as the intrusion prevention system will not be able to inspect the encrypted data.
40+
However, not only does HTTPS introduce some overhead due to the encryption/decryption, it also makes caching in forward proxies impossible.
41+
Therefore, it is strongly discouraged to use HTTPS as default.
3042

31-
Installing a Stratum 1 requires a GEO API license key, which will be used to find the (geographically) closest Stratum 1 server for your client and proxies.
32-
More information on how to (freely) obtain this key is available in the CVMFS documentation: https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup.
43+
### Manual configuration
3344

34-
You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`.
45+
In order to set up a Stratum 1 manually, you can make use of the instructions in the [Private Stratum 1 replica server](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/access/stratum1/)
46+
section of the MultiXscale tutorial ["Best Practices for CernVM-FS in HPC"](https://multixscale.github.io/cvmfs-tutorial-hpc-best-practices/).
3547

36-
Furthermore, the Stratum 1 runs a Squid server. The template configuration file can be found at `templates/eessi_stratum1_squid.conf.j2`.
37-
If you want to customize it, for instance for limiting the access to the Stratum 1, you can make your own version of this template file
38-
and point to it by setting `local_stratum1_cvmfs_squid_conf_src` in `inventory/local_site_specific_vars.yml`.
39-
See the comments in the example file for more details.
48+
### Configuration using Ansible
4049

41-
Start by installing Ansible:
50+
The recommended way for setting up an EESSI Stratum 1 is by running the Ansible playbook `stratum1.yml`
51+
from the [filesystem-layer repository on GitHub](https://github.com/EESSI/filesystem-layer).
52+
For the commands in this section, we are assuming that you cloned this repository, and your working directory is `filesystem-layer`.
53+
54+
!!! note GEO API
55+
Installing a Stratum 1 usually requires a GEO API license key, which will be used to find the (geographically) closest Stratum 1 server for your client and proxies.
56+
However, for a private Stratum 1 this can be skipped, and you can disable the use of the GEO API in the configuration of your clients by setting `CVMFS_USE_GEOAPI=no`.
57+
In this case, they will just connect to your local Stratum 1 by default.
58+
59+
If you do want to set up the GEO API, you can find more information on how to (freely) obtain this key in the CVMFS documentation: https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup.
60+
61+
You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`.
62+
63+
Start by installing Ansible, e.g.:
4264

4365
```bash
4466
sudo yum install -y ansible
@@ -47,128 +69,65 @@ sudo yum install -y ansible
4769
Then install Ansible roles for EESSI:
4870

4971
```bash
50-
ansible-galaxy role install -r requirements.yml -p ./roles --force
72+
ansible-galaxy role install -r ./requirements.yml --force
5173
```
5274

53-
Make sure you have enough space in `/srv` (on the Stratum 1) since the snapshot of the Stratum 0
54-
will end up there by default. To alter the directory where the snapshot gets copied to you can add
55-
this variable in `inventory/host_vars/<url-or-ip-to-your-stratum1>`:
56-
75+
Make sure you have enough space in `/srv` on the Stratum 1, since the snapshots of the repositories
76+
will end up there by default. To alter the directory where the snapshots get stored you can manually
77+
create a symlink before running the playbook:
5778
```bash
58-
cvmfs_srv_mount: /srv
79+
sudo ln -s /lots/of/space/cvmfs /srv/cvmfs
5980
```
6081

61-
Make sure that you have added the hostname or IP address of your server to the
62-
`inventory/hosts` file. Finally, install the Stratum 1 using one of the two following options.
82+
Also make sure that you have added the hostname or IP address of your server to the
83+
`inventory/hosts` file, that you are able to log in to the server from the machine that is going to run the playbook
84+
(preferably using an SSH key), and that you can use `sudo`.
6385

64-
Option 1:
86+
Finally, install the Stratum 1 using:
6587

6688
``` bash
67-
# -b to run as root, optionally use -K if a sudo password is required
68-
ansible-playbook -b [-K] -e @inventory/local_site_specific_vars.yml stratum1.yml
89+
# -b to run as root, optionally use -K if a sudo password is required, and optionally include your site-specific variables
90+
ansible-playbook -b [-K] [-e @inventory/local_site_specific_vars.yml] stratum1.yml
6991
```
70-
71-
Option2:
72-
73-
Create a ssh key pair and make sure the `ansible-host-keys.pub` is in the
74-
`$HOME/.ssh/authorized_keys` file on your Stratum 1 server.
75-
76-
```bash
77-
ssh-keygen -b 2048 -t rsa -f ~/.ssh/ansible-host-keys -q -N ""
78-
```
79-
80-
Then run the playbook:
81-
82-
```bash
83-
ansible-playbook -b --private-key ~/.ssh/ansible-host-keys -e @inventory/local_site_specific_vars.yml stratum1.yml
84-
```
85-
8692
Running the playbook will automatically make replicas of all the repositories defined in `group_vars/all.yml`.
8793

8894

89-
## Step 2: request a firewall exception
90-
91-
(This step is not implemented yet and can be skipped)
92-
93-
You can request a firewall exception rule to be added for your Stratum 1 server by
94-
[opening an issue on the GitHub page of the filesystem layer repository](https://github.com/EESSI/filesystem-layer/issues/new).
95+
### Verification of the Stratum 1 using `curl`
9596

96-
Make sure to include the IP address of your server.
97-
98-
## Step 3: Verification of the Stratum 1
99-
100-
When the playbook has finished your Stratum 1 should be ready. In order to test your Stratum 1, even
101-
without a client installed, you can use `curl`.
97+
When the playbook has finished, your Stratum 1 should be ready. In order to test your Stratum 1,
98+
even without a client installed, you can use `curl`:
10299

103100
```bash
104101
curl --head http://<url-or-ip-to-your-stratum1>/cvmfs/software.eessi.io/.cvmfspublished
105102
```
106-
This should return:
103+
This should return something like:
107104

108105
```bash
109106
HTTP/1.1 200 OK
110107
...
111-
X-Cache: MISS from <url-or-ip-to-your-stratum1>
112-
```
113-
114-
The second time you run it, you should get a cache hit:
115-
116-
```bash
117-
X-Cache: HIT from <url-or-ip-to-your-stratum1>
118-
108+
Content-Type: application/x-cvmfs
119109
```
120110

121-
Example with the Norwegian Stratum 1:
111+
Example with the EESSI Stratum 1 running in AWS:
122112

123113
```bash
124-
curl --head http://bgo-no.stratum1.cvmfs.eessi-infra.org/cvmfs/software.eessi.io/.cvmfspublished
114+
curl --head http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmfspublished
125115
```
126116

127-
You can also test access to your Stratum 1 from a client, for which you will have to install the CVMFS
128-
[client](https://github.com/EESSI/filesystem-layer#clients).
129-
130-
Then run the following command to add your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file:
117+
### Verification of the Stratum 1 using a CVMFS client
131118

132-
```bash
133-
echo 'CVMFS_SERVER_URL="http://<url-or-ip-to-your-stratum1>/cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi-hpc.org.local
134-
```
119+
You can, of course, also test access to your Stratum 1 from a client.
120+
This requires you to install a CernVM-FS client and add the Stratum 1 to the client configuration;
121+
this is explained in more detail on the [native installation page](../getting_access/native_installation.md).
135122

136-
If this is the first time you set up the client you now run:
137-
138-
```bash
139-
sudo cvmfs_config setup
140-
```
141-
142-
If you already had configured the client before, you can simply reload the config:
143-
144-
```bash
145-
sudo cvmfs_config reload -c software.eessi.io
146-
```
147-
148-
Finally, verify that the client connects to your new Stratum 1 by running:
123+
Then verify that the client connects to your new Stratum 1 by running:
149124

150125
```bash
151126
cvmfs_config stat -v software.eessi.io
152127
```
153128

154-
Assuming that your new Stratum 1 is the geographically closest one to your client, this should return:
129+
Assuming that your new Stratum 1 is working properly, this should return something like:
155130

156131
```bash
157132
Connection: http://<url-or-ip-to-your-stratum1>/cvmfs/software.eessi.io through proxy DIRECT (online)
158133
```
159-
160-
161-
## Step 4: request an EESSI DNS name
162-
163-
In order to keep the configuration clean and easy, all the EESSI Stratum 1 servers have a DNS name
164-
`<your site>.stratum1.cvmfs.eessi-infra.org`, where `<your site>` is often a short name or
165-
abbreviation followed by the country code (e.g. `rug-nl` or `bgo-no`). You can request this for
166-
your Stratum 1 by mentioning this in the issue that you created in Step 2, or by opening another
167-
issue.
168-
169-
## Step 5: include your Stratum 1 in the EESSI configuration
170-
171-
If you want to include your Stratum 1 in the EESSI configuration, i.e. allow any (nearby) client to be able to use it,
172-
you can open a pull request with updated configuration files. You will only have to add the URL to your Stratum 1 to the
173-
`urls` list of the `eessi_cvmfs_server_urls` variable in the
174-
[`all.yml` file](https://github.com/EESSI/filesystem-layer/blob/main/inventory/group_vars/all.yml).

‎docs/getting_access/native_installation.md

+52-6
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Native installation
22

3+
## Installation for single clients
4+
35
Setting up native access to EESSI, that is a system-wide deployment that does not require workarounds like
46
[using a container](eessi_container.md), requires the installation and configuration of [CernVM-FS](https://cernvm.cern.ch/fs).
57

@@ -62,14 +64,58 @@ The good news is that all of this only requires a handful commands :astonished:
6264
sudo cvmfs_config setup
6365
```
6466

67+
## Installation for larger systems (e.g. clusters)
68+
69+
When using CernVM-FS on a larger number of local clients, e.g. on a HPC cluster or set of workstations,
70+
it is very strongly recommended to at least set up some Squid proxies close to your clients.
71+
These Squid proxies will be used to cache content that was recently accessed by your clients,
72+
which reduces the load on the Stratum 1 servers and reduces the latency for your clients.
73+
As a rule of thumb, you should use about one proxy per 500 clients, and have a minimum of two.
74+
Instructions for setting up a Squid proxy can be found in the [CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-squid.html) and
75+
in the [CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/#32-setting-up-a-proxy).
76+
77+
Additionally, setting up a private Stratum 1, which will make a full copy of the repository,
78+
can be beneficial to improve the latency and bandwidth even further, and to be better protected against network outages.
79+
Instructions for setting up your own EESSI Stratum 1 can be found in [setting up your own CernVM-FS Stratum 1 mirror server](../filesystem_layer/stratum1.md).
80+
81+
### Configuring your client to use a Squid proxy
82+
83+
If you have set up one or more Squid proxies, you will have to add them to your CernVM-FS client configuration.
84+
This can be done by removing `CVMFS_CLIENT_PROFILE="single"` from `/etc/cvmfs/default.local`, and add the following line:
85+
86+
```
87+
CVMFS_HTTP_PROXY="http://ip-of-your-1st-proxy:port|http://ip-of-your-2nd-proxy:port"
88+
```
89+
90+
In this case, both proxies are equally preferable.
91+
More advanced use cases can be found in [the CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-list-examples).
92+
93+
### Configuring your client to use a private Stratum 1 mirror server
94+
95+
If you have set up your own Stratum 1 mirror server that replicates the EESSI CernVM-FS repositories,
96+
you can instruct your CernVM-FS client(s) to use it by prepending your newly created Stratum 1 to the existing list of EESSI Stratum 1 servers by creating a local CVMFS configuration file for the EESSI domain:
97+
98+
```bash
99+
echo 'CVMFS_SERVER_URL="http://<url-or-ip-to-your-stratum1>/cvmfs/@fqrn@;$CVMFS_SERVER_URL"' | sudo tee -a /etc/cvmfs/domain.d/eessi.io.local
100+
```
101+
65102
!!! note
103+
By prepending your new Stratum 1 to the list of existing Stratum 1 servers, your clients should by default use the private Stratum 1.
104+
In case of downtime of your private Stratum 1, they will also still be able to make use of the public EESSI Stratum 1 servers.
105+
106+
107+
### Applying changes in the CernVM-FS client configuration files
108+
109+
After you have made any changes to the CernVM-FS client configuration, you will have to apply them.
110+
If this is the first time you set up the client, you can simply run:
66111

67-
:point_up: The commands above only cover the basic installation of EESSI.
112+
```bash
113+
sudo cvmfs_config setup
114+
```
68115

69-
This is good enough for an individual client, or for testing purposes,
70-
but for a production-quality setup you should also set up a Squid proxy cache.
116+
If you already had configured the client before, you can reload the configuration for the EESSI repository (or, similarly, for any other repository) using:
71117

72-
For large-scale systems, like an HPC cluster, you should also consider setting up your own CernVM-FS Stratum-1 mirror server.
118+
```bash
119+
sudo cvmfs_config reload -c software.eessi.io
120+
```
73121

74-
For more details on this, please refer to the
75-
[*Stratum 1 and proxies section* of the CernVM-FS tutorial](https://cvmfs-contrib.github.io/cvmfs-tutorial-2021/03_stratum1_proxies/).

0 commit comments

Comments
 (0)