Reporting fuzzer failures

[Porges]

At the moment if a fuzzer exits for some unknown reason we don't do anything special, and the fuzzer will be restarted.

At the very least we should report the exit code information, and we also have had requests (#2313) to capture mini/core dumps from the fuzzer process.

We will need some way to report this information to the users, so I'm opening up this discussion to get some ideas.

We probably will want to report the information via the API, maybe something along the lines of:

unexpected_exits:
- timestamp: "…"
  exit_code: 12345
  dump_file: "https://…"
  run_log: "https://…" # run-specific stderr or stdout log

[stishkin]

From my end - would be nice to separate OneFuzz task failures (in general as well as fuzzer specific failures) into subcomponents.
Plus have info weather this failure is user-actionable or not: bug in the fuzzer or in OneFuzz

• task failure due to "fuzzer" failure
• task failure due to "agent" failure

[chkeita]

I propose to use the Heartbeat mechanism. We could add new message type here and emit it when we see task failure

[tevoinea]

I think Onefuzz failure vs user failure is important because you might want different information in each situation. Also, some way to get node information around the time of failure like ram usage, cpu usage, syslog of the last 30 seconds prior to crash.

We don't even need to necessarily capture all of that information at the time of the crash, for example we could probably query cpu usage from azure on the service side after we get the crash event.