getting 403 for kubernetes_logs from kube apiserver | AWS EKS

[burlesque13]

Hello,

I am trying to deploy vector helm chart in AWS EKS.

Installing via the command from Readme:
helm install vector vector/vector -n vector -f values.yaml

There is a custom config in the values.yaml:

customConfig:
   data_dir: /vector-data-dir
   sources:
     kubernetes_logs:
       type: kubernetes_logs
       self_node_name: ${VECTOR_SELF_NODE_NAME:-unspecified}
     internal_metrics:
       type: internal_metrics
   sinks:
     s3_archives:
       inputs:
         - kubernetes_logs
       type: aws_s3
       auth:
         access_key_id: "ACCESS_KEY_ID"
         secret_access_key: "SECRET_ACCESS_KEY"
       region: us-east-1
       bucket: vector-k8s-logs
       key_prefix: date=%Y-%m-%d
       compression: gzip
       encoding: ndjson

S3 connection is OK. But I got the 403 error for kubernetes_logs component from kube apiserver:

2022-02-02T15:10:32.392470Z DEBUG source{component_kind=“source” component_id=kubernetes_logs component_type=kubernetes_logs component_name=kubernetes_logs}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://10.100.0.1:443/api/v1/namespaces?&allowWatchBookmarks=true&resourceVersion=0&timeoutSeconds=290&watch=true method=GET version=HTTP/1.1 headers={“authorization”: Sensitive, “user-agent”: “Vector/0.19.0 (x86_64-unknown-linux-gnu da60b55 2021-12-28)“, “accept-encoding”: “identity”} body=[empty]
2022-02-02T15:10:32.398595Z DEBUG source{component_kind=“source” component_id=kubernetes_logs component_type=kubernetes_logs component_name=kubernetes_logs}:http: vector::internal_events::http_client: HTTP response. status=403 Forbidden version=HTTP/1.1 headers={“audit-id”: “a1aef5f6-0f7c-4324-98f6-93714f318598”, “cache-control”: “no-cache, private”, “content-type”: “application/json”, “x-content-type-options”: “nosniff”, “date”: “Wed, 02 Feb 2022 15:10:32 GMT”, “content-length”: “291”} body=[291 bytes]

What am I missing in the configuration and how to resolve the 403 error?
The k8s cluster consists of 6 nodes, maybe the ${VECTOR_SELF_NODE_NAME} variable is wrong? Please advise.

Thank you in advance.

[spencergilbert]

The current design of the chart doesn't deploy any RBAC resources unless you've deployed as an Agent (link). This is because it doesn't really make sense to use the kubernetes_logs source with an aggregator style deployment as Vector needs to be running on each node for that source to work properly.

I'd suggest setting role: "Agent" in your values (or --set role=Agent in your install command), which would include the linked RBAC resources needed to make requests to the API server. If you really need the source running on the aggregator roles feel free to open an issue at vectordotdev/helm-charts to discuss.

[burlesque13]

Just tested the chart with --set role=Agent - it works as expected.

Very grateful for the answer!