scalyr-agent-2-config --export-config should resolve symbol links

[schra]

What

The instructions from https://app.scalyr.com/help/scalyr-agent-k8s#export-config don't work. scalyr-agent-2-config --export-config doesn't export the configuration such that I could customize it.

$ kubectl -n scalyr exec scalyr-84g88 -- scalyr-agent-2-config --export-config - | tail -n +2 | tar -xz
$ cat agent.d/agent.json
cat: agent.d/agent.json: No such file or directory

Why

Files under agent.d/ are symbol links to non-existent files (files that are not part of the tar archive):

$ find
.

$ kubectl -n scalyr exec scalyr-84g88 -- scalyr-agent-2-config --export-config - | tail -n +2 | tar -xz

$ find
.
./agent.json
./agent.d
./agent.d/scalyr-server.json
./agent.d/docker.json
./agent.d/k8s_events.json
./agent.d/agent.json

$ ls -lah
total 24K
drwxrwxr-x  3 andre andre 4,0K Jan 19 17:04 .
drwxrwxrwt 33 root  root   12K Jan 19 17:03 ..
drwxrwxr-x  2 andre andre 4,0K Jan 19 17:04 agent.d
-rw-rw-r--  1 andre andre  471 Aug 10 16:20 agent.json

$ ls -lah agent.d/
total 8,0K
drwxrwxr-x 2 andre andre 4,0K Jan 19 17:04 .
drwxrwxr-x 3 andre andre 4,0K Jan 19 17:04 ..
lrwxrwxrwx 1 andre andre   17 Jan 19 15:50 agent.json -> ..data/agent.json
lrwxrwxrwx 1 andre andre   18 Jan 19 15:50 docker.json -> ..data/docker.json
lrwxrwxrwx 1 andre andre   22 Jan 19 15:50 k8s_events.json -> ..data/k8s_events.json
lrwxrwxrwx 1 andre andre   25 Jan 19 15:50 scalyr-server.json -> ..data/scalyr-server.json

$ cat agent.d/*
cat: agent.d/agent.json: No such file or directory
cat: agent.d/docker.json: No such file or directory
cat: agent.d/k8s_events.json: No such file or directory
cat: agent.d/scalyr-server.json: No such file or directory

We are also using the official (unmodified) Docker image and only customize via ConfigMaps.

Docker image version: 2.1.15

Definition of done

Resolve symbol links when exporting the configuration files via scalyr-agent-2-config --export-config

[weilliu]

@schra I think you may have a syntax error on your export command. Could you try the following command and see if this works for you?

kubectl -n scalyr exec -i <scalyr-agent pod>-- scalyr-agent-2-config --export-config - | tar -xz

This command successfully exports all config files from the k8s agent pod. I don't see the symbolic links you were observing in your environment.

Weilis-MacBook-Pro:temp weililiu$ ls -lah agent.d/
total 32
drwxr-xr-x  6 weililiu  staff   204B Jan 19 15:51 .
drwxr-xr-x  4 weililiu  staff   136B Jan 19 15:51 ..
-rw-r--r--  1 weililiu  staff    70B Jan 16 13:40 api-key.json
-rw-r--r--  1 weililiu  staff   103B Jan 16 13:40 docker.json
-rw-r--r--  1 weililiu  staff   109B Jan 16 13:40 k8s_events.json
-rw-r--r--  1 weililiu  staff   128B Jan 16 13:40 scalyr-server.json
Weilis-MacBook-Pro:temp weililiu$ cat agent.d/*
{
  import_vars: [ "SCALYR_API_KEY" ],
  api_key: "$SCALYR_API_KEY"
}
{
  "monitors":[
    {
      "module": "scalyr_agent.builtin_monitors.kubernetes_monitor",
    }
  ]
}
{
  "monitors":[
    {
      "module": "scalyr_agent.builtin_monitors.kubernetes_events_monitor"
    }
  ]
}
{
  "import_vars": [ { "var": "SCALYR_SERVER", "default": "https://agent.scalyr.com" } ],
  "scalyr_server": "$SCALYR_SERVER"
}

Your original command "cat agent.d/agent.json" returns nothing because agent.json doesn't reside inside of the agent.d directory. I test it with agent v2.1.17, but I don't think the version makes any difference here. Could you give it another try with the above kubctl export command?

[schra]

@weilliu

Could you try the following command and see if this works for you?

Sure:

$ kubectl get pods -n scalyr
NAME           READY   STATUS    RESTARTS   AGE
scalyr-bbpq8   1/1     Running   0          23m
scalyr-vld67   1/1     Running   0          25m

Because you wanted to run your exact command: there is a whitespace mising

$ kubectl -n scalyr exec -i scalyr-bbpq8-- scalyr-agent-2-config --export-config - | tar -xz
Error: unknown flag: --export-config
See 'kubectl exec --help' for usage.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Now run the command again but with the whitespace. See #704 why this command fails (this is another bug)

$ kubectl -n scalyr exec -i scalyr-bbpq8 -- scalyr-agent-2-config --export-config - | tar -xz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Again, see #704 for the workaround command that I'll now use

$ kubectl -n scalyr exec -i scalyr-vld67 -- scalyr-agent-2-config --export-config - | tail -n +2 | tar -xz

$ ls -lah agent.d/
total 8,0K
drwxrwxr-x 2 andre andre 4,0K Jan 21 13:52 .
drwxrwxr-x 3 andre andre 4,0K Jan 21 13:52 ..
lrwxrwxrwx 1 andre andre   17 Jan 21 13:14 agent.json -> ..data/agent.json
lrwxrwxrwx 1 andre andre   18 Jan 21 13:14 docker.json -> ..data/docker.json
lrwxrwxrwx 1 andre andre   22 Jan 21 13:14 k8s_events.json -> ..data/k8s_events.json
lrwxrwxrwx 1 andre andre   25 Jan 21 13:14 scalyr-server.json -> ..data/scalyr-server.json

I don't see the symbolic links you were observing in your environment.

But did you also use a ConfigMap that mounted the files into the pod? Our ConfigMap mounts to /etc/scalyr-agent-2/agent.d. Using ConfigMaps is what you also recommend in your documentation.

Your original command "cat agent.d/agent.json" returns nothing because agent.json doesn't reside inside of the agent.d directory.

Well, we have two agent.json's in our setup. We have agent.json and agent.d/agent.json.

[weilliu]

@schra Sorry about missing the white space in the sample command. I think #704 is a bug on 2.1.15 which I can't reproduce on 2.1.17. I will create a bug report for the engineering to fix it.

Regards to the configMap setting, the recommended way is to add the additional configurations to the scalyr-config or specify them in a different configMap (ex. scalyr-config-agent-d).

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: scalyr-config-agent-d
          mountPath: /etc/scalyr-agent-2/agent.d
      volumes:
        - name: scalyr-config-agent-d
          configMap:
            name: scalyr-config-agent-d

The above configuration is an example from our documentation. Is this similar to what you have?
You can then add the content to each config files to the scalyr-config-agent-d configMap.

$ kubectl get configmap scalyr-config-agent-d -o yaml
apiVersion: v1
data:
  api-key.json: |+
        {
          "import_vars": [ "SCALYR_API_KEY" ],
          "api_key": "$SCALYR_API_KEY"
        }
  docker.json: |+
        {
          "monitors":[
            {
              "module": "scalyr_agent.builtin_monitors.kubernetes_monitor",
            }
          ]
        }
  scalyr-server.json: |+
        {
          "import_vars": [ { "var": "SCALYR_SERVER", "default": "https://agent.scalyr.com" } ],
          "scalyr_server": "$SCALYR_SERVER"
        }
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-07-17T12:11:46Z"
  name: scalyr-config-agent-d
  namespace: default
  resourceVersion: "15331074"
  selfLink: /api/v1/namespaces/default/configmaps/scalyr-config-agent-d
  uid: 0be7c62f-a88c-11e9-a6bc-8a0ac2b88958

This is much easier than mounting the pod configs to local files. The export function is mainly designed for the files that were initially edited inside of containers, and later users need to export them to local disk for reusing the same configs to bake a custom Scalyr agent pod.

[schra]

Sorry about missing the white space in the sample command. I think #704 is a bug on 2.1.15 which I can't reproduce on 2.1.17. I will create a bug report for the engineering to fix it.

No worries, and thank you :)

Regards to the configMap setting, the recommended way is to add the additional configurations to the scalyr-config or specify them in a different configMap (ex. scalyr-config-agent-d).

The above configuration is an example from our documentation. Is this similar to what you have?
You can then add the content to each config files to the scalyr-config-agent-d configMap.

This exactly what we do in our deployment. But k8s seems to mount the files from the configmap weirdly because they seem to be symbol links.

[weilliu]

@schra I was able to reproduce the same result, but I do think that this is expected behavior. Exporting agent config files helps you create a configMap scalyr-config-agent-d.

To modify Scalyr Agent config files on the fly, first extract a running agent's configuration into a ConfigMap. Then modify that ConfigMap and inject it back by saving the ConfigMap.

https://app.scalyr.com/help/scalyr-agent-k8s#export-config

Since you already had the configMap created, you should be able to edit the configMap directly with kubectl edit configMap scalyr-config-agent-d -n scalyr.

It seems that there is a workaround solution by adding subpath. You could give it a try if it is really needed.
https://stackoverflow.com/questions/62776362/k8s-configmap-mounted-inside-symbolic-link-to-data-directory