Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration from loki-distributed to loki fails when using umbrella chart #15878

Open
MatusJenca2 opened this issue Jan 22, 2025 · 0 comments
Open

Comments

@MatusJenca2
Copy link

Describe the bug
I tried to migrate from loki-distributed to loki in our monitoring-stack using this guide

However i got this error.

helm upgrade --install monitoring ~/kubernetes-monitoring-stack/chart -f loki/values-loki-migration.yaml --reuse-values --dependency-update 
Error: template: dnation-kubernetes-monitoring-stack/charts/loki/templates/write/statefulset-write.yaml:89:20: executing "dnation-kubernetes-monitoring-stack/charts/loki/templates/write/statefulset-write.yaml" at <include "loki.image" .>: error calling include: template: dnation-kubernetes-monitoring-stack/charts/loki-distributed/templates/_helpers.tpl:78:33: executing "loki.image" at <.global.registry>: nil pointer evaluating interface {}.registry
helm.go:86: 2025-01-22 12:38:47.333622711 +0100 CET m=+0.542132745 [debug] template: dnation-kubernetes-monitoring-stack/charts/loki/templates/write/statefulset-write.yaml:89:20: executing "dnation-kubernetes-monitoring-stack/charts/loki/templates/write/statefulset-write.yaml" at <include "loki.image" .>: error calling include: template: dnation-kubernetes-monitoring-stack/charts/loki-distributed/templates/_helpers.tpl:78:33: executing "loki.image" at <.global.registry>: nil pointer evaluating interface {}.registry

I suspect that this is related to this issue in helm as both loki and loki-distributed charts use loki.image variable in templates and to quote Helm docs:

Defined templates (templates created inside a {{ define }} directive) are globally accessible. That means that a chart and all of its subcharts will have access to all of the templates created with {{ define }}.
For that reason, all defined template names should be namespaced.

Because we use an ubrella chart, there's a conflict. I don't see any way around this other than code change, that's why I am creating this isssue.
To Reproduce
Steps to reproduce the behavior:

  1. Create a subchart with both loki and loki-distributed as dependencies i.e. our monitoring-stack.
  2. Install with loki-distributed enabled and loki disabled - This is current state
  3. Follow migration guide
  4. Deploy with both subcharts enabled as per the guide (step 1)

Expected behavior
Both loki and loki distributed are deployed, we may continue to step 2 of the guide

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output
values before migration

kube-prometheus-stack:
  kubeScheduler:
    serviceMonitor:
      insecureSkipVerify: true

  kubeControllerManager:
    serviceMonitor:
      insecureSkipVerify: true
# I tried to manually enter global.registry wherever needed, still got the same error
global:
  registry: https://hub.docker.com/
loki:
  enabled: false
loki-distributed:
  global:
    registry: https://hub.docker.com/
  registry: https://hub.docker.com/
  enabled: true
# loki config omitted, may contain sensitive information

values-loki-migration.yaml

# Drop logs from loki 
promtail:
  config:
    snippets:
      common:
      - source_labels:
        - "__meta_kubernetes_pod_label_app_kubernetes_io_component"
      regex: "(canary|read|write)"
      action: "drop"
loki2:
  registry: https://hub.docker.com/
  global:
    registry: https://hub.docker.com/
  enabled: true
  migrate:
    fromDistributed:
      enabled: true
      memberlistService: monitoring-loki-memberlist
  auth_enabled: true
  minio:
    enabled: false
  loki:
    storage:
      bucketNames:
        chunks: monitoring-loki-chunks  # an example bucket name for chunks
        ruler: monitoring-loki-ruler  # an example bucket name for rules
      s3:
        endpoint: https://minio:[email protected]/loki
        s3ForcePathStyle: true
    limits_config:
      retention_period: 0  # infinite retention
    compactor:
    retention_enabled: false # infinite retention
global:
  registry: https://hub.docker.com/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant