[Security Solution] Remove non-errors and user errors from console lo…

…gs (#206973)

## Summary

This PR will drastically reduce the number of console logs from the
Detection Engine in the overview cluster. If your team is depending on
the `INFO`, `WARNING`, or `verification_exception` or `ml job missing`
`ERROR` logs in that cluster, please raise your concerns here.

Addresses elastic/kibana-team#1395,
elastic/kibana-team#1333

Historically Detection Rules have written an enormous amount of data to
the console logs. This was helpful in debugging years ago before we had
the event log, but now we generally don't use them much. The console
logs all still get scooped up and sent to the overview cluster though.
Every rule execution writes two or more status changes (first to
'running', then to the final status later on) and these go to the
console, the event log, and the rule SO. The end result is 76% of all
logs are coming from detection rules changing status, mostly successful
statuses. These provide little value on their own.

This PR restricts console logging from detection rules to only non-user
errors. User errors and execution statuses below the error level will be
logged in the console at the `debug` level. "Unexpected" errors like
search exceptions, timeouts, etc will still appear as errors in the
console logs. The general idea is that the logs from detection rules in
the console should represent some kind of unexpected system failure.

To implement this change, I updated the console logging logic in both
the security rule execution logger. User errors reported to the
framework will still create console error logs since they're logged at
the framework level.

## Testing
Create rules that run and generate warnings (e.g. missing index), user
errors (EQL verification exceptions), and non-user errors. An easy way
to create a non-user error at the moment is running a threshold rule
when at least one index searched maps `@timestamp` as a `keyword`
instead of `date`. The non user errors still show up in console logs as
errors. User errors and warnings only show up as debug logs and with
debug logging enabled in the Kibana config.

x-pack/solutions/security/plugins/security_solution/common/api/detection_engine/rule_monitoring/model/log_level.ts

@@ -53,7 +53,7 @@ export const logLevelFromNumber = (num: number | null | undefined): LogLevel =>
   return LogLevelEnum.error;
 };
 
-export const logLevelFromExecutionStatus = (status: RuleExecutionStatus): LogLevel => {
+export const eventLogLevelFromExecutionStatus = (status: RuleExecutionStatus): LogLevel => {
   switch (status) {
     case RuleExecutionStatusEnum['going to run']:
     case RuleExecutionStatusEnum.running:
@@ -68,3 +68,13 @@ export const logLevelFromExecutionStatus = (status: RuleExecutionStatus): LogLev
       return LogLevelEnum.trace;
   }
 };
+
+export const consoleLogLevelFromExecutionStatus = (
+  status: RuleExecutionStatus,
+  userError?: boolean
+): LogLevel => {
+  if (!userError && status === RuleExecutionStatusEnum.failed) {
+    return LogLevelEnum.error;
+  }
+  return LogLevelEnum.debug;
+};

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/client_for_executors/client.ts

@@ -19,7 +19,7 @@ import type {
   LogLevel,
 } from '../../../../../../../common/api/detection_engine/rule_monitoring';
 import {
-  logLevelFromExecutionStatus,
+  consoleLogLevelFromExecutionStatus,
   LogLevelSetting,
   logLevelToNumber,
   RuleExecutionStatusEnum,
@@ -159,7 +159,7 @@ export const createRuleExecutionLogClientForExecutors = (
   const writeStatusChangeToConsole = (args: NormalizedStatusChangeArgs, logMeta: ExtMeta): void => {
     const messageParts: string[] = [`Changing rule status to "${args.newStatus}"`, args.message];
     const logMessage = messageParts.filter(Boolean).join('. ');
-    const logLevel = logLevelFromExecutionStatus(args.newStatus);
+    const logLevel = consoleLogLevelFromExecutionStatus(args.newStatus, args.userError);
     writeMessageToConsole(logMessage, logLevel, logMeta);
   };
 

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_monitoring/logic/rule_execution_log/event_log/event_log_writer.ts

@@ -10,7 +10,7 @@ import type { IEventLogService } from '@kbn/event-log-plugin/server';
 import { SAVED_OBJECT_REL_PRIMARY } from '@kbn/event-log-plugin/server';
 import type { LogLevel } from '../../../../../../../common/api/detection_engine/rule_monitoring';
 import {
-  logLevelFromExecutionStatus,
+  eventLogLevelFromExecutionStatus,
   logLevelToNumber,
   ruleExecutionStatusToNumber,
 } from '../../../../../../../common/api/detection_engine/rule_monitoring';
@@ -107,7 +107,7 @@ export const createEventLogWriter = (eventLogService: IEventLogService): IEventL
     },
 
     logStatusChange: (args: StatusChangeArgs): void => {
-      const logLevel = logLevelFromExecutionStatus(args.newStatus);
+      const logLevel = eventLogLevelFromExecutionStatus(args.newStatus);
       eventLogger.logEvent({
         '@timestamp': nowISO(),
         message: args.message,