Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO_PROXY variable not honored #1372

Open
waldner opened this issue Jul 20, 2022 · 18 comments
Open

NO_PROXY variable not honored #1372

waldner opened this issue Jul 20, 2022 · 18 comments

Comments

@waldner
Copy link

waldner commented Jul 20, 2022

Running fluentd-kubernetes-daemonset:v1.14.6-debian-elasticsearch7-1.0 inside a k8s cluster behind a proxy. The container has the HTTP_PROXY/HTTPS_PROXY (and their lowercase version) variables set, which are honored; however, the proxy should not be used to connect to the k8s API, so I set NO_PROXY (and no_proxy) to kubernetes,10.43.0.1,kubernetes.default.svc, yet it looks like the API is not being accessed directly. Here are some errors from the log:

Successfully installed fluent-plugin-kubernetes-objects-1.1.12
1 gem installed
2022-07-20 13:13:02 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes-objects' version '1.1.12'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-07-20 13:13:02 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-07-20 13:13:02 +0000 [info]: gem 'fluentd' version '1.14.6'
2022-07-20 13:13:02 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
#<Thread:0x00007fe581a3da90 run> terminated with exception (report_on_exception is true):
/fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:87:in `rescue in start_pod_watch': start_pod_watch: Exception encountered setting up pod watch from Kubernetes API v1 endpoint https://10.43.0.1:443/api: pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope ({"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \\"system:serviceaccount:myns:default\\" cannot list resource \\"pods\\" in API group \\"\\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403} (Fluent::ConfigError)
)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:78:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:130:in `rescue in handle_exception': pods is forbidden: User "system:serviceaccount:myns:default" cannot list resource "pods" in API group "" at the cluster scope (Kubeclient::HttpError)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:120:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'
/fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response': 403 Forbidden (RestClient::Forbidden)
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:836:in `process_result'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
	from /usr/local/lib/ruby/2.7.0/net/http.rb:933:in `start'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/rest-client-2.1.0/lib/restclient/resource.rb:51:in `get'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:352:in `block in get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:121:in `handle_exception'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:350:in `get_entities'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:224:in `block (2 levels) in define_entity_methods'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/kubeclient-4.9.3/lib/kubeclient/common.rb:101:in `method_missing'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:102:in `get_pods_and_start_watcher'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:79:in `start_pod_watch'
	from /fluentd/vendor/bundle/ruby/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.5/lib/fluent/plugin/kubernetes_metadata_watch_pods.rb:32:in `set_up_pod_thread'

On another cluster running without the proxy (everything else the same), no error is produced.

@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Oct 19, 2022
@waldner
Copy link
Author

waldner commented Oct 19, 2022

The issue is still relevant.

@github-actions github-actions bot removed the stale label Oct 20, 2022
@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Jan 19, 2023
@waldner
Copy link
Author

waldner commented Jan 19, 2023

The issue is still relevant.

@github-actions github-actions bot removed the stale label Jan 20, 2023
@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Apr 21, 2023
@waldner
Copy link
Author

waldner commented Apr 21, 2023

The issue is still relevant.

@github-actions github-actions bot removed the stale label Apr 22, 2023
@mikaelkrief
Copy link

any solution to this issue ?

@vittico
Copy link

vittico commented Jul 19, 2023

Is there any solution available?

@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Oct 18, 2023
@waldner
Copy link
Author

waldner commented Oct 18, 2023

This is still an issue.

@github-actions github-actions bot removed the stale label Oct 19, 2023
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Jan 17, 2024
@waldner
Copy link
Author

waldner commented Jan 17, 2024

This is still an issue.

@github-actions github-actions bot removed the stale label Jan 18, 2024
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Apr 17, 2024
@waldner
Copy link
Author

waldner commented Apr 17, 2024

This is still an issue.

@github-actions github-actions bot removed the stale label Apr 18, 2024
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Jul 17, 2024
@waldner
Copy link
Author

waldner commented Jul 17, 2024

This is still an issue.

@github-actions github-actions bot removed the stale label Jul 18, 2024
Copy link

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

@github-actions github-actions bot added the stale label Oct 17, 2024
@waldner
Copy link
Author

waldner commented Oct 19, 2024

This is still an issue.

@github-actions github-actions bot removed the stale label Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants