Argo CDからGrafana Dashboardの更新が行われない問題 #1536

iaoiui · 2022-02-18T13:19:16Z

Grafana Dashboardをjsonで管理しArgo CD経由でDeployすると、下記の警告とエラーが発生し正常にダッシュボードが更新されていない状態となっている。

t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[default private]"
99
t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[default private]"
98
t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[private default]"
97
t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
96
t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
95
t=2022-02-18T13:09:39+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[default private]"
94
t=2022-02-18T13:09:39+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
93
t=2022-02-18T13:09:39+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
92
t=2022-02-18T13:09:39+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
91
t=2022-02-18T13:09:39+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
90
t=2022-02-18T13:09:39+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
89
t=2022-02-18T13:09:49+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
88
t=2022-02-18T13:09:49+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
87
t=2022-02-18T13:09:59+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
86
t=2022-02-18T13:09:59+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
85
t=2022-02-18T13:10:09+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
84
t=2022-02-18T13:10:09+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF

The text was updated successfully, but these errors were encountered:

iaoiui · 2022-02-18T13:21:37Z

grafana/grafana#41085 (comment)

disableDeletion: false
allowUiUpdates: true

iaoiui · 2022-02-18T13:34:56Z

Provisioning Dashboardの仕様
https://grafana.com/docs/grafana/latest/administration/provisioning/#reusable-dashboard-urls

If the dashboard in the JSON file contains an UID, Grafana forces insert/update on that UID. This allows you to migrate dashboards between Grafana instances and provisioning Grafana from configuration without breaking the URLs given because the new dashboard URL uses the UID as identifier. When Grafana starts, it updates/inserts all dashboards available in the configured folders. If you modify the file, then the dashboard is also updated. By default, Grafana deletes dashboards in the database if the file is removed. You can disable this behavior using the disableDeletion setting.

Dashboard用のJSONがUIDを含んでいる場合、そのUIDを持つ既に存在するDashboardを置き換える。
JSONファイルを変更するとDashboardは更新されるが、disableDeletion を使うことでDashboardの削除を防ぐことができる。

iaoiui · 2022-02-18T13:36:31Z

現状のdisableDeletion 設定はgrafana.yaml内で行なっておりtrueとなっている。
そのため、削除無効化された状態となりArgo CDによる置き換えがうまくいっていないと考えられる。

iaoiui · 2022-02-24T10:57:10Z

allowUIUpdatesの有効化を行なっても該当の事象が改善していない

t=2022-02-24T10:51:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[private default]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[default private]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[default private]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[private default]"
t=2022-02-24T10:51:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:51:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:51:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:51:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:51:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:51:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[default private]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[private default]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
t=2022-02-24T10:51:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:51:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:51:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:51:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:51:58+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:51:58+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[default private]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[default private]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[private default]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[private default]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[default private]"
t=2022-02-24T10:52:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:52:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:52:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:52:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[public default private]"
t=2022-02-24T10:52:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:52:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:52:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:52:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:52:58+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:52:58+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:53:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:53:08+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[default private]"
t=2022-02-24T10:53:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:53:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:53:18+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:53:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:53:18+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:53:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:53:28+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:53:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:53:38+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=HdeaRHxWz times=2 providers="[private default]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=tmsOtSxZk times=2 providers="[private default]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=fRIvzUZMz times=2 providers="[default private]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=JABGX_-mz times=2 providers="[default private]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=peR80gTGk times=2 providers="[default private]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=N4HUPtT6z times=3 providers="[default private public]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=AWSBillig times=2 providers="[private default]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="the same UID is used more than once" logger=provisioning.dashboard uid=4XuMd2Iiz times=2 providers="[default private]"
t=2022-02-24T10:53:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=default
t=2022-02-24T10:53:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=private
t=2022-02-24T10:53:48+0000 lvl=warn msg="dashboards provisioning provider has no database write permissions because of duplicates" logger=provisioning.dashboard provider=public
t=2022-02-24T10:53:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Contour-HTTProxy.json error=EOF
t=2022-02-24T10:53:48+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=default file=/var/lib/grafana/dashboards/default/Kubernetes-Pod.json error=EOF

oke-py · 2022-02-24T11:19:33Z

sqlite3を使えるエフェメラルコンテナを突っ込んで/var/lib/grafana/dashboards/とsqliteの中身を見てみようかと

oke-py · 2022-02-24T18:25:08Z

むーん

kubectl debug -it grafana-0 --image=nouchka/sqlite3 --target=grafana-0
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource").

oke-py · 2022-02-25T14:05:19Z

provisioned dashboardをフォルダー移動できるようになった（例 General => Public => deprecated）。削除はできないまま。

@iaoiui PRにFixと書いてあるからClosedになりましたが、以下の理由でReopenしてよいですよね？

prdには反映していない
エラーログは変化ないように見える
意図したフォルダー分けになっていない

oke-py · 2022-02-25T14:06:51Z

8.4に上げたら部分的には解決するかも？

Provisioning: Fix duplicate validation when multiple organizations have been configured.
https://github.com/grafana/grafana/releases/tag/v8.4.0

grafana/grafana#43530

oke-py · 2022-02-27T01:22:12Z

k exec -it grafana-0 -c grafana -- ls -lR /var/lib/grafana/dashboards/

/var/lib/grafana/dashboards/:
total 8
drwxrwsr-x    2 grafana  472           4096 Jan 15 03:47 default
drwxrwsr-x    2 grafana  472           4096 Feb 12 17:43 private
drwxrwsr-x    2 grafana  472             37 Feb 12 09:14 public

/var/lib/grafana/dashboards/default:
total 344
-rw-rw-r--    1 grafana  472          11927 Feb 12 11:30 AWS Billing.json
-rw-rw-r--    1 grafana  472          24280 Dec 18 16:38 AWS CloudWatch Synthetics.json
-rw-rw-r--    1 grafana  472          41172 Feb 12 11:30 Amazon EC2.json
-rw-rw-r--    1 grafana  472          47626 Dec  3 14:49 Amazon RDS-2.json
-rw-rw-r--    1 grafana  472          37847 Feb 12 11:30 Amazon RDS.json
-rw-rw-r--    1 grafana  472          33222 Feb 12 11:30 Cluster Monitoring for Kubernetes.json
-rwxrwxr-x    1 grafana  472              0 Nov 27 00:28 Contour-HTTProxy.json
-rw-rw-r--    1 grafana  472          59238 Feb 12 11:30 Kubernetes Cluster (Prometheus).json
-rwxrwxr-x    1 grafana  472              0 Nov 27 00:28 Kubernetes-Pod.json
-rw-rw-r--    1 grafana  472          23904 Feb 12 11:30 Logging Dashboard via Loki.json
-rw-rw-r--    1 grafana  472          37268 Feb 12 11:30 Persistent Volume Usage.json
-rw-rw-r--    1 grafana  472          15227 Feb 12 11:30 o11y2022 proposals.json

/var/lib/grafana/dashboards/private:
total 332
-rw-rw-r--    1 grafana  472          11927 Feb 27 00:34 AWS Billing.json
-rw-rw-r--    1 grafana  472          41172 Feb 27 00:34 Amazon EC2.json
-rw-rw-r--    1 grafana  472          37847 Feb 27 00:34 Amazon RDS.json
-rw-rw-r--    1 grafana  472          33222 Feb 27 00:34 Cluster Monitoring for Kubernetes.json
-rwxrwxr-x    1 grafana  472          23882 Feb 27 00:34 Contour-HTTProxy.json
-rw-rw-r--    1 grafana  472          59238 Feb 27 00:34 Kubernetes Cluster (Prometheus).json
-rwxrwxr-x    1 grafana  472          15081 Feb 27 00:34 Kubernetes-Pod.json
-rw-rw-r--    1 grafana  472          23904 Feb 27 00:34 Logging Dashboard via Loki.json
-rw-rw-r--    1 grafana  472          37268 Feb 27 00:34 Persistent Volume Usage.json
-rw-rw-r--    1 grafana  472          20362 Feb 27 00:34 o11y2022 main.json
-rw-rw-r--    1 grafana  472          15227 Feb 12 17:43 o11y2022 proposals.json

/var/lib/grafana/dashboards/public:
total 16
-rw-rw-r--    1 grafana  472          15239 Feb 27 00:34 o11y2022 proposals.json

oke-py · 2022-02-27T01:34:14Z

ひたすら削除しました。

k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/AWS\ Billing.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/AWS\ CloudWatch\ Synthetics.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Amazon\ EC2.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Amazon\ RDS-2.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Amazon\ RDS.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Cluster\ Monitoring\ for\ Kubernetes.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Contour-HTTProxy.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Kubernetes\ Cluster\ \(Prometheus\).json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Kubernetes-Pod.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Logging\ Dashboard\ via\ Loki.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Persistent\ Volume\ Usage.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/o11y2022\ proposals.json

k exec -it grafana-0 -c grafana -- ls -lR /var/lib/grafana/dashboards/

/var/lib/grafana/dashboards/:
total 4
drwxrwsr-x    2 grafana  472              6 Feb 27 01:29 default
drwxrwsr-x    2 grafana  472           4096 Feb 27 01:24 private
drwxrwsr-x    2 grafana  472             37 Feb 12 09:14 public

/var/lib/grafana/dashboards/default:
total 0

/var/lib/grafana/dashboards/private:
total 316
-rw-rw-r--    1 grafana  472          11927 Feb 27 00:34 AWS Billing.json
-rw-rw-r--    1 grafana  472          41172 Feb 27 00:34 Amazon EC2.json
-rw-rw-r--    1 grafana  472          37847 Feb 27 00:34 Amazon RDS.json
-rw-rw-r--    1 grafana  472          33222 Feb 27 00:34 Cluster Monitoring for Kubernetes.json
-rwxrwxr-x    1 grafana  472          23882 Feb 27 00:34 Contour-HTTProxy.json
-rw-rw-r--    1 grafana  472          59238 Feb 27 00:34 Kubernetes Cluster (Prometheus).json
-rwxrwxr-x    1 grafana  472          15081 Feb 27 00:34 Kubernetes-Pod.json
-rw-rw-r--    1 grafana  472          23904 Feb 27 00:34 Logging Dashboard via Loki.json
-rw-rw-r--    1 grafana  472          37268 Feb 27 00:34 Persistent Volume Usage.json
-rw-rw-r--    1 grafana  472          20362 Feb 27 00:34 o11y2022 main.json

/var/lib/grafana/dashboards/public:
total 16
-rw-rw-r--    1 grafana  472          15239 Feb 27 00:34 o11y2022 proposals.json

oke-py · 2022-02-27T01:36:13Z

Podを再作成しました。

k rollout restart sts grafana

解決した感あります。

oke-py · 2022-02-27T01:40:21Z

deprecatedフォルダーに移動しておいたダッシュボードをUI上で削除できた。

AWS CloudWatch Synthetics
Amazon RDS

それに伴い、deprecatedフォルダーを削除しました。

oke-py · 2022-02-27T05:33:44Z

prdで実行しました。

k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/AWS\ CloudWatch\ Synthetics.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Amazon\ EC2.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Amazon\ RDS.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Cluster\ Monitoring\ for\ Kubernetes.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Contour-HTTProxy.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Kubernetes\ Cluster\ \(Prometheus\).json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Kubernetes-Pod.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Logging\ Dashboard\ via\ Loki.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/Persistent\ Volume\ Usage.json
k exec -it grafana-0 -c grafana -- rm /var/lib/grafana/dashboards/default/o11y2022\ proposals.json

k exec -it grafana-0 -c grafana -- ls -lR /var/lib/grafana/dashboards/

/var/lib/grafana/dashboards/:
total 4
drwxrwsr-x    2 grafana  472              6 Feb 27 05:32 default
drwxrwsr-x    2 grafana  472           4096 Feb 14 07:32 private
drwxrwsr-x    2 grafana  472             63 Feb 13 10:03 public

/var/lib/grafana/dashboards/default:
total 0

/var/lib/grafana/dashboards/private:
total 296
-rw-rw-r--    1 grafana  472          11927 Feb 20 22:45 AWS Billing.json
-rw-rw-r--    1 grafana  472          41172 Feb 20 22:45 Amazon EC2.json
-rw-rw-r--    1 grafana  472          37847 Feb 20 22:45 Amazon RDS.json
-rw-rw-r--    1 grafana  472          33222 Feb 20 22:45 Cluster Monitoring for Kubernetes.json
-rw-r--r--    1 root     472          23889 Feb 20 22:44 Contour-HTTProxy.json
-rw-rw-r--    1 grafana  472          59238 Feb 20 22:45 Kubernetes Cluster (Prometheus).json
-rw-r--r--    1 root     472          15088 Feb 20 22:44 Kubernetes-Pod.json
-rw-rw-r--    1 grafana  472          23904 Feb 20 22:45 Logging Dashboard via Loki.json
-rw-rw-r--    1 grafana  472          37268 Feb 20 22:45 Persistent Volume Usage.json

/var/lib/grafana/dashboards/public:
total 40
-rw-rw-r--    1 grafana  472          20810 Feb 20 22:45 o11y2022 main.json
-rw-rw-r--    1 grafana  472          15248 Feb 20 22:45 o11y2022 proposals.json

k rollout restart sts grafana

iaoiui · 2022-03-05T02:31:25Z

下記2つの設定をした後も、直ちに事象は解決しなかったがDB内のダッシュボード定義削除による暫定対処方法を確立できたため、一旦クローズする。
下記2つの設定を入れても事象が再現した場合は、Grafana8.4へのマイナーアップデートによる事象解決を検討する。

disableDeletion: false
allowUiUpdates: true

b1gb4by · 2023-02-27T04:16:54Z

Grafanaのアップデートもしたので再度検証を行うため、オープンにします、

iaoiui added the bug Something isn't working label Feb 18, 2022

iaoiui self-assigned this Feb 18, 2022

iaoiui mentioned this issue Feb 18, 2022

ダッシュボードの削除を有効化 #1537

Merged

7 tasks

oke-py closed this as completed in #1537 Feb 19, 2022

iaoiui mentioned this issue Feb 24, 2022

change allowUiUpdates false to true #1544

Merged

7 tasks

iaoiui reopened this Feb 24, 2022

iaoiui assigned oke-py Feb 24, 2022

iaoiui mentioned this issue Feb 25, 2022

allowUiUpdatesをtrueに変更 #1553

Merged

7 tasks

oke-py closed this as completed in #1553 Feb 25, 2022

oke-py reopened this Feb 25, 2022

This was referenced Feb 25, 2022

Grafana Dashboardの明示的なuid指定を削除 #1554

Merged

mainブランチのダッシュボードjsonを参照するよう変更 #1557

Merged

oke-py mentioned this issue Feb 27, 2022

dev: o11y2022 main ダッシュボードをpublicフォルダーに移す #1558

Merged

7 tasks

iaoiui closed this as completed Mar 5, 2022

b1gb4by mentioned this issue Jul 6, 2022

o11y2022のダッシュボード無効化 cloudnativedaysjp/observability#61

Closed

b1gb4by reopened this Feb 27, 2023

b1gb4by mentioned this issue Feb 28, 2023

Disable "disableDeletion" for Dev #2393

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Argo CDからGrafana Dashboardの更新が行われない問題 #1536

Argo CDからGrafana Dashboardの更新が行われない問題 #1536

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 24, 2022

oke-py commented Feb 24, 2022

oke-py commented Feb 24, 2022

oke-py commented Feb 25, 2022

oke-py commented Feb 25, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022 •

edited

Loading

oke-py commented Feb 27, 2022 •

edited

Loading

iaoiui commented Mar 5, 2022

b1gb4by commented Feb 27, 2023

Argo CDからGrafana Dashboardの更新が行われない問題 #1536

Argo CDからGrafana Dashboardの更新が行われない問題 #1536

Comments

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 18, 2022

iaoiui commented Feb 24, 2022

oke-py commented Feb 24, 2022

oke-py commented Feb 24, 2022

oke-py commented Feb 25, 2022

oke-py commented Feb 25, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022

oke-py commented Feb 27, 2022 • edited Loading

oke-py commented Feb 27, 2022 • edited Loading

iaoiui commented Mar 5, 2022

b1gb4by commented Feb 27, 2023

oke-py commented Feb 27, 2022 •

edited

Loading

oke-py commented Feb 27, 2022 •

edited

Loading