Profiler: wrapper zap logger for datadog tracer agent

[nathantournant]

What does this PR do?

• Adds new functionality
• Alters existing functionality
• Fixes a bug
• Improves documentation or testing

Please briefly describe your changes as well as the motivation behind them:

• The datadog logger interface is simple but did not fit the zap logger we used. This provides a wrapper that puts all tracer/profiler logs in our usual zap SugaredLogger, providing the usual tags / logging context, and very basic info/warn/error level parsing. The parsing allows to not receive tracer/profiler startup logs (and warnings) as errors, but set to the appropriate levels.

Code Quality Checklist

• The documentation is up to date.
• My code is sufficiently commented and passes continuous integration checks.
• I have signed my commit (see Contributing Docs).

Testing

• I leveraged continuous integration testing
  • by depending on existing unit tests or end-to-end tests.
  • by adding new unit tests or end-to-end tests.
• I manually tested the following steps:
  • Raw log example
  • SugaredLogger example
  • locally.
  • as a canary deployment to a cluster.

[datadog-datadog-prod-us1]

Datadog Report

Branch report: nathan/CHAOS-653/fix-profiler-setup
Commit report: ccdbd4f

✅ chaos-controller: 0 Failed, 0 New Flaky, 477 Passed, 0 Skipped, 2m 50.44s Wall Time

[datadog-datadog-prod-us1]

Datadog Report

Branch report: nathan/CHAOS-653/fix-profiler-setup
Commit report: 6160b9e

❄️ chaos-controller: 0 Failed, 3 New Flaky, 483 Passed, 0 Skipped, 5m 36.8s Wall Time

New Flaky Tests (3)

• [It] Cache Handler verify events sent should not fire any warning event on disruption - Controller Suite - Last Failure

  Expand for error

   Timed out after 35.232s.
   Expected success, but got an error:
       <*errors.StatusError | 0xc0006741e0>: 
       disruptions.chaos.datadoghq.com "ee77f2bb-649a-4605-9396-a40d79172006-lqqwp" not found
       {
           ErrStatus: {
               TypeMeta: {Kind: "", APIVersion: ""},
               ListMeta: {
                   SelfLink: "",
                   ResourceVersion: "",
   ...
  

• [It] Disruption Controller disruption expires naturally should target all the selected pods - Controller Suite - Last Failure

  Expand for error

   Timed out after 73.087s.
   Expected success, but got an error:
       <*errors.StatusError | 0xc0003254a0>: 
       disruptions.chaos.datadoghq.com "f31090f4-7555-4e41-b102-188112321550-xv8zd" not found
       {
           ErrStatus: {
               TypeMeta: {Kind: "", APIVersion: ""},
               ListMeta: {
                   SelfLink: "",
                   ResourceVersion: "",
   ...
  

• [It] Disruption Controller Injection Statuses disruption expired statuses PreviouslyInjected is reached after PausedInjected and stays until disruption expires - Controller Suite - Last Failure

  Expand for error

   Timed out after 59.554s.
   Expected success, but got an error:
       <*errors.StatusError | 0xc000b526e0>: 
       disruptions.chaos.datadoghq.com "4f6c9f49-e300-492a-bccc-567bf38e1ca9-78qxg" not found
       {
           ErrStatus: {
               TypeMeta: {Kind: "", APIVersion: ""},
               ListMeta: {
                   SelfLink: "",
                   ResourceVersion: "",
   ...
  

[taihuynh167]

string matching can quickly become inefficient and also prone to cAse-mismatched.

• how does the tracer work? Is it a separate process that will consume our existing logs?
• does this prevent our existing logs from being published?

[nathantournant]

• The datadog tracer interface for logging uses and requires one and only one function; logger.Log(msg string). No levels, despite having levels in the message. While string matching is basic, it's simple enough and will allow us to have the tracer log at the expected level using our regular logger. If you think we might miss errors with this parsing, I'd be happy to move the default to error in case the format changes or the parsing breaks.

• The tracer works as a single instance across all packages. It emits non-critical but useful logs from times to times, such as the Startup Log, and some warnings/errors when it can't reach the agent. At creation, it has an option to use a given logger we input. I would like to use our regular Zap contextualised logger for this, to make those logs have the same context as every other logs we have.

• No it doesn't; currently our logging is the same as usual, but logs created by the Tracer lack some of the context and formatting our regular logs have. They also all appear as errors.