System architecture questions

[lars-t-hansen]

We're starting to have a lot of moving parts and we're going to need to integrate them at some point. See also #19 #53 #57. Summarizing:

• sonar runs (by cron) on the ML nodes and writes output to files on a disk, call it D, which is available from the ML nodes.
• sonar will ditto run on every node on Fox and write output to a disk, call it F, available from Fox.
• (and maybe more systems)
• sonalyze runs (manually, by cron to run bughunt/cpuhog scripts, or by naicreport to produce load data; naicreport is in turn run by cron) on some machine that has access to D and/or F. D and F could be the same or separate disks. sonalyze could run multiple places on different logs.
• some of those runs of sonalyze are a little expensive and it may or may not be a good idea to run them on the HPC/ML hardware directly
• the raw output data produced by sonalyze and naicreport are consumed by email systems and web servers
• in particular a web server needs to serve data produced by naicreport

Questions:

• where do we store and how do we share log data (shared disk or some kind of rsync)?
• where do we run the various scripts and programs that process log data and generate other data?
• where do we run the web server that will serve some of the generated data?
• are these places the same for the ml nodes and fox?

There are probably many more questions about the use of the system, so this is just the start of a conversation.

[lars-t-hansen]

The questions have been answered:

• we store and share (at least for the ml nodes) on /itf-fi-ml/shared/users/sonarmon, probably
• scripts run on moneypenny
• the web server runs on a vm with a static, public ip
• we'll get to fox when we get to it, but i believe there could be a completely parallel setup for fox without this impacting any of the infrastructure at all