Change log level on the fly without restart

[szvincze]

Overview

Currently the log level can be configured via environment variable NSM_LOG_LEVEL which means the process will use the configured value for its whole lifetime. It is fine in most of the cases, but when one would like to change the log level because closer monitoring is needed, it would be expected to be able to do the change without restarting the process that would be monitored. In the vast majority of the cases the restart makes impossible the immediate monitoring action, because after the configuration change we should initiate restart and then just wait for the issue to happen again.

For temporary setting it would be good if the log level could be changed via sending IPC signals to the process. For example, if we would like to monitor forwarder-vpp then we send SIGUSR1 to the process and it changes its log level to TRACE, when we captured the needed logs the original log level can be restored by sending signal SIGUSR2.

[arp-est]

Hi,
We also ha a reference implementation, if this signalling approach is okay with you guys

networkservicemesh/cmd-forwarder-vpp#1171

[Ex4amp1e]

Possible solution:

1. Add term "NSM config"
2. Move LOG_LEVEL configuring from ENV config to NSM config
3. [Optional] In future - move other ENVs

Algorithm for LOG_LEVEL:

1. Every cmd repo will contain structure, that we will read data from file
2. On every request there will be check if the file has been modified, in positive case - read new value and update LOG_LEVEL in structure
3. In scope k8s deployment - we will store data in configmap and mount it into each pod as file

[denis-tingaikin]

cc @edwarnicke

[arp-est]

Hi, did the prs for the cmds, here's a list:
networkservicemesh/cmd-cluster-info-k8s#157
networkservicemesh/cmd-forwarder-ovs#420
networkservicemesh/cmd-forwarder-sriov#778
networkservicemesh/cmd-ipam-vl3#221
networkservicemesh/cmd-lb-vl3-vpp#107
networkservicemesh/cmd-map-ip-k8s#246
networkservicemesh/cmd-nsc-init#785
networkservicemesh/cmd-nsc#655
networkservicemesh/cmd-nsc-simple-docker#256
networkservicemesh/cmd-nsc-vpp#723
networkservicemesh/cmd-nse-firewall-vpp#541
networkservicemesh/cmd-nse-icmp-responder#621
networkservicemesh/cmd-nse-icmp-responder-vpp#705
networkservicemesh/cmd-nse-l7-proxy#226
networkservicemesh/cmd-nse-remote-vlan#249
networkservicemesh/cmd-nse-simple-vl3-docker#271
networkservicemesh/cmd-nse-vfio#538
networkservicemesh/cmd-nse-vl3-vpp#368
networkservicemesh/cmd-nsmgr-proxy#523
networkservicemesh/cmd-nsmgr#710
networkservicemesh/cmd-registry-memory#689
networkservicemesh/cmd-registry-proxy-dns#670

I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?

networkservicemesh/cmd-exclude-prefixes-k8s#319
networkservicemesh/cmd-nse-supplier-k8s#355
networkservicemesh/cmd-registry-k8s#481

I think these are all the cmds where the log level change would be applicable.

[denis-tingaikin]

Hello @arp-est ,

Many thanks for doing this.

I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?

Yeah, we have faced a problem in SDK K8s, and those components were not updated. Now that the problem is resolved, could you please rebase these PRs?

[arp-est]

Hello @arp-est ,

Many thanks for doing this.

I have some problems with the prs below, the reference to the sdk is old there, how do you guys usually update the versions?

Yeah, we have faced a problem in SDK K8s, and those components were not updated. Now that the problem is resolved, could you please rebase these PRs?

Right away

[arp-est]

Hi, I rebased it but in those prs its not the sdk-k8s, but the regular sdk, that is missing the SetupLevelChangeOnSignal function, I see only the version of sdk-k8s was updated in the go.mod file.

[denis-tingaikin]

It seems like it's done.

Many thanks, @arp-est.

Created a ticket to add an integration test for this feature: #12522.

Closing this one.