Deploy sonar prototype to Fox

[lars-t-hansen]

Since sonar and sonalyze now seem to be OK for multi-node systems, we should start collecting data on Fox.

There are some issues around whether Sonar is fast enough, which I'm addressing:

At the moment a Sonar invocation on a Fox node takes about 100ms (this is not a high quality measurement); as NordicHPC/sonar#86 shows, we should be able to cut this time by roughly half. Deploying to Fox probably does not depend on that fix, but it would be nice to get it done.

We would like to also implement the other three features for performance, reliability, and quality reasons, but it would be good to first measure their impact on Fox nodes with and without GPUs.

• Read data directly from /proc NordicHPC/sonar#86
• Add a pre-check whether host/node is GPU-aware before running any other GPU commands NordicHPC/sonar#44
• Read data directly from GPU APIs NordicHPC/sonar#87
• Use exec, not shell, to run subprograms NordicHPC/sonar#88

In addition, there's the deployment checklist:

• Where do the binaries go (shared is OK for now)
• Where do the data go (shared is OK for now)
• Where do the lockfiles go (want a node-local file system)
• How do we run sonar (cron or something else)
• Under what UID do we run sonar (want something that is common between nodes and analysis)
• If cron, are there existing cronfiles on the system (for that UID) that must work?
• ec-fox-sw is not allowed to run crontab
• probably a directory for sonar inside ~ec-fox-sw for sonar cron file and anything else it needs, eg scripts?
• Where do we run the analysis jobs
• How do we ensure that all the jobs stay up all the time
• What will the config files and scripts look like
• Names of uploaded reports should not be generic #248
• Guard against multiple concurrent invocations NordicHPC/sonar#123
• Cleanup lockfiles on reboot
• sonalyze load produces NaN values on gpu-less systems for rgpu, rgpumem #253
• Dashboard must not assume the presence of GPU data #254
• Bring sonar up on all the Fox nodes

[lars-t-hansen]

If cron is not possible, it may be possible to trigger the sonar run remotely by ssh, every few minutes, for every node. Anyway we'll find it out.

[Sabryr]

I think for the Fox meeting, better to propose that you would deploy this on 4 compute nodes (2 with GPU and 2 without). i.e. not all at the same time. Even on NRIS side (consultation with Radovan et all) we ask "some" nodes for the fist round. This gives the admins more assurance on a production system.

[lars-t-hansen]

Some results from a meeting

• We run sonar and analysis as ec-fox-sw (actually this turns out to be controversial)
• We use cron on each node but we will do this by copying a file into /etc/cron.d/ (also on each node) and make the system run it, not have a crontab running for the user
• We run analysis on the interactive nodes, login-2 was mentioned, but eventually we move this into a VM (which will be the target of message-based, not disk-based, logging)
• Sonar data currently go into /cluster/var/sonar/data
• Lockfiles should go in /var/tmp per bhm
• The sonar binary can go into /cluster/sbin (or with sonalyze/naicreport) for now, though eventually we want it in /node/bin, but that requires more infra and we want to be more in production than we currently are
• The sonalyze/naicreport binaries can be anywhere really, perhaps in home dir for user running the analysis, /fp/home01/u01/ec-fox-sw/sonar/bin
• Crontab sources live in /fp/home01/u01/ec-fox-sw/sonar/cron (and of course github)
• Probably the dashboard will show a host as down if the scripts stop running (b/c heartbeat is gone) and we don't need any more than that

[lars-t-hansen]

The analysis is live on Fox (on a small number of nodes).

[lars-t-hansen]

The analysis is now live on all Fox nodes.

[lars-t-hansen]

Lockfile cleanup can be accomplished by an @reboot action in the cron file.

[lars-t-hansen]

The analysis is live on Fox cpu, gpu, interactive, and login nodes. Data are exfiltrated to the remote analysis host. All blocking bugs are really sonar bugs.

[lars-t-hansen]

Lockfile removal: ~sonar is /var/run/sonar on compute and gpu nodes, and this would work just fine. But on int and login it is /home/sonar - not fine. Have petitioned to change homedir to /var/run/sonar on int and login and then just change the script from using /var/tmp to using /var/run/sonar for lockfiles.

Edit: Discussion with fox admins: We'll move the homedir on the int and login nodes, it was always the intention that it should have been /var/run/sonar, and if new nodes are created they will get that too.

Edit^2: Except then we need a service to create /var/run/sonar on boot, probably, one can do something via tmpfiles.d(5) but it's becoming fairly elaborate, esp if we want to move toward making sonar a systemd service anyway.

[lars-t-hansen]

I'm kicking the lockfile issue down the road, issue #352.