Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug with hyphens/dash (-) on queue names #324

Open
thiarthur opened this issue Oct 3, 2024 · 1 comment
Open

Bug with hyphens/dash (-) on queue names #324

thiarthur opened this issue Oct 3, 2024 · 1 comment

Comments

@thiarthur
Copy link

I have an old project using micro-services architecture, where the queues are named "{service-name}-module." When inspecting the metrics endpoints, some metrics work properly:

# TYPE celery_active_consumer_count gauge
celery_active_consumer_count{queue_name="data-manager-search"} 1.0
celery_active_consumer_count{queue_name="data-manager"} 1.0
celery_active_consumer_count{queue_name="data-manager-storage"} 1.0
# HELP celery_active_worker_count The number of active workers in broker queue.
# TYPE celery_active_worker_count gauge
celery_active_worker_count{queue_name="data-manager-search"} 1.0
celery_active_worker_count{queue_name="data-manager"} 1.0
celery_active_worker_count{queue_name="data-manager-storage"} 1.0
# HELP celery_active_process_count The number of active processes in broker queue.
# TYPE celery_active_process_count gauge
celery_active_process_count{queue_name="data-manager-search"} 24.0
celery_active_process_count{queue_name="data-manager"} 24.0
celery_active_process_count{queue_name="data-manager-storage"} 24.0

However, for other metrics, the queue is shown as "celery" for queue_name:

# HELP celery_task_received_total Sent when the worker receives a task.
# TYPE celery_task_received_total counter
celery_task_received_total{hostname="data-manager-worker-58664599dd-6pnc9",name="data_manager.tasks.sleeper_task",queue_name="celery"} 6.0
celery_task_received_total{hostname="data-manager-worker-548d9d96d6-vxnzv",name="data_manager.tasks.sleeper_task",queue_name="celery"} 1.0

To test, I created a simple task:

@app.task(queue="data-manager")
def sleeper_task(sleep_time: int):
    import time

    time.sleep(sleep_time)
    return {"status": "ok"}

This is the Celery command I use in my Kubernetes cluster:

containers:
    - name: data-manager-worker
      image: data-manager:latest
      imagePullPolicy: IfNotPresent
      command:
          [
              "celery",
              "-A",
              "data_manager.tasks",
              "worker",
              "--loglevel=INFO",
              "--task-events",
              "-Q",
              "data-manager-storage,data-manager-search,data-manager",
          ]

After troubleshooting, I found that using underscores (_) in the queue names worked properly, while using hyphens (-) caused the issue.
metrics.txt

I am unsure whether this issue originates from the celery-exporter or from the way Celery events handle queue names, but the tasks execute successfully with both queue naming conventions.

If there's any other information that i can provide, let me know.

@danihodovic
Copy link
Owner

Can you enable the debug log on the exporter and paste the logs here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants