Configuration guide of structured logging for Kyuubi server

[pan3793]

Why are the changes needed?

It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in KafkaAppender and JSON Template Layout, thus the goal could be achieved by just a few configurations.

To simplify the user setup steps, this PR adds log4j-layout-template-json-<version>.jar into Kyuubi binary tarball.

PS: I also plan to support sending engine bootstrap process(e.g. spark-submit) logs into Kafka with specific labels in the follow-up PRs.

How was this patch tested?

Manually test.

Configuration in $KYUUBI_HOME/conf/log4j2.xml

<Configuration status="INFO">
  <Appenders>
    <Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
      <JsonTemplateLayout>
        <EventTemplateAdditionalField key="app" value="kyuubi"/>
        <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
        <EventTemplateAdditionalField key="host" value="${hostName}"/>
      </JsonTemplateLayout>
      <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
      <Property name="compression.type" value="gzip"/>
    </Kafka>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="kafka"/>
    </Root>
  </Loggers>
</Configuration>

Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

For reviewers who want to verify or play with this feature, awesome-kyuubi/hadoop-testing provides an out-of-box setup via the following rendering command

ansible-playbook build.yaml -e "{grafana_enabled: true, prometheus_enabled: true, loki_enabled: true, kafka_enabled: true, kafka_ui_enabled: true}"

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (d50cf17) to head (9556da2).
Report is 3 commits behind head on master.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6861   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         687     687           
  Lines       42473   42473           
  Branches     5796    5796           
======================================
  Misses      42473   42473           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Thanks, merged to master