WatchTower Metrics: Add

[taw123]

Is your feature request related to a problem? Please describe.

First I wanted to thank you for all your work on WatchTower, it seems quite feature rich.

I recently migrated here to WT from Ouroboros, and noticed per your documentation that your metrics support is currently listed as "experimental" and hoped you might consider taking on some of the metrics I recently lost in moving off Ouroboros. Specifically interested in seeing SPECIFIC container names so I can log at what was updated, how many time, and when. Ideally I would take that data and table in a version (if you don't consider vending that as well (at least for containers you update) since you could easily harvest it when you pull/check digest).

FYI this is a sample Dashboard from Ouroboros in case a pic is worth a thousand words.

second item is potentially a bug or perhaps a request for documentation update as the http endpoint doesn't function as expected but DOES sometimes(?) actually DOES produce a valid endpoint/url just no data on it.

configuration/documentation describes EXPERIMENTAL support for http metric scraping. I had originally thought the endpoint was broken as I see no CURRENT data in the endpoint (even wrote up a bug to post)....but it seems OCASSIONAL some dat must be getting produced as Prometheus does have data in it from the endpoint that as I mention still SHOWS noting on the endpoint when I hit it from the browser. Very strange.

I enabled WATCHTOWER_HTTP_API_METRICS, set a password with WATCHTOWER_HTTP_API_TOKEN and finally re-mapped your internal port (for the http server- 8080) to 9789 as port was in use in my deployment.

Without WatchTower container I get no response at the endpoint URL, and with WatchTower up I can connect to the URL's root (http://192.168.42.84:9789) and get the expected: 404 page not found, HOWEVER connecting to the documented metrics url-(http://192.168.42.84:9789/v1/metrics) produces ABSOLUTELY NOTHING with the container loaded and updates having successfully occurred overnight.

NOTE:
I am NOT using the HTTP API to TRIGGER updates, and the documentation doesn't state that it is actually REQUIRED for metrics export, if that is NOT the case and thus perhaps thee reason for my problem it should be clearly stated in the Metrics section and ideally revised to also for use without requiring HTTP triggered updates...

Relevant compose segment:

  Watchtower:             # Auto update containers replace (Ouroboros)  github.com/containrrr/watchtower/ -- Wiki: containrrr.dev/watchtower/
    image: containrrr/watchtower:latest
    container_name: Watchtower
    hostname: Watchtower.local
    restart: unless-stopped  
    environment:
      PUID: 1004              
      PGID: 100
      TZ: America/Los_Angeles
      WATCHTOWER_DISABLE_CONTAINERS: QNAP-HomeBridge vscode_stecky FileBot-XPRA MariaDB
      WATCHTOWER_INCLUDE_STOPPED: True          # Update even if container stopped
      WATCHTOWER_REVIVE_STOPPED: False  #             ... but don't start it if it WAS stopped before the update
      WATCHTOWER_CLEANUP: True
      WATCHTOWER_SCHEDULE: "0 0 2 * * *"    # run at 2 am  Cron format, but instead of five fields, it uses six, adding one at the beginning for the seconds
      WATCHTOWER_NO_STARTUP_MESSAGE: true      # for **debug**
      WATCHTOWER_LOG_LEVEL: info       # panic, fatal, error, warn, [info], debug or trace (trace is most excessive and shows confidential data)
      WATCHTOWER_LOG_FORMAT: Pretty      # [Auto], LogFmt, Pretty or JSON

      WATCHTOWER_HTTP_API_METRICS: True                             # Enables a metrics endpoint, exposing prometheus metrics via HTTP
      WATCHTOWER_HTTP_API_TOKEN: ### not-telling ###        # <String> in Prometheus scrape job
      
 ##                  Implement hooks/scripts INSIDE each container to orchestrate container update/restart dependencies -- https://containrrr.dev/watchtower/lifecycle-hooks/
##      WATCHTOWER_LIFECYCLE_HOOKS: True
              
    networks:
      monitor:
    ports:
      - 9789:8080            # Map to external port 9789 from inside native web/prometheus port 8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json       # Pass GitHub & DockerHub account auth/generated tokens 

It SEEMS like the endpoint doesn't CONTINUOUSLY vend Metric data???

Perhaps I am misunderstanding the function/design, or perhaps there is indeed a bug here and should file bug report? Please advice.

Describe the solution you'd like

Ideally a continuous stream/persistance of metrics at the endpoint that I can hit directly/with Prometheus.

Second would love to work with you on adding additional metrics described above.

Describe alternatives you've considered

Ouroboros

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[Lorax66]

I believe you misunderstood the documentation (the same way I did).
By adapting the command from the "http api mode" documentation (curl -H "Authorization: Bearer mytoken" localhost:8080/v1/metrics), I get the data, that can be fed to prometheus to get pretty graphs.

I think the documentation is misleading on that part and would like to see it improved.

[taw123]

@Lorax66 Thanks for the follow up, and apologies for not noticing it sooner.

I tried manually curl'ing as you suggest but to not avail. I get the same results (no data) per below.

# curl -H "Authorization: myPassword" localhost:9789/v1/metrics
# 

or even using the REAL HOST ip:

# curl -H "Authorization: myPassword" 192.168.42.84:9789/v1/metrics
#

The WatchTower container is running of course on a NAT'd IP own the same host I am running the curl (and Prometheus on). I mapped the container's native port to 9789 so what I entered SHOULD be equivalent to your suggestion.

Perhaps I am misunderstanding more than just the doc/wiki?

I am ASSUMING that the path should CONTINUOUSLY have content available for scrape so long as the container is running (well and have run at least once to have content to view...) like all other jobs I have in the past integrated (perhaps I am wrong?)

While this is only Prometheus exporters I have ever used that required authentication it's seems that possibly the authentication is (silently) failing? Since typing in the same curl with an INCORRECT token results in the same lack of data returned...

# curl -H "Authorization: myINCORECTpassword" localhost:9789/v1/metrics
# 

So right token or wrong, I get the same lack of data from the API. If I try a port that I KNOW doesn't have a listener on it I get thee expected:

curl: (7) Failed to connect to 192.168.42.84 port 85 after 0 ms: Connection refused

I then drew the conclusion that it SEEMS likely that there IS indeed SOMETHING listening on the port/IP I am attempting to use, and therefore either some kind of Auth problem (unfortunately that seems to be a silent error response) or a misunderstanding of what the API is supposed to do since it seems you have this working 👍

Regardless of my earlier mistake which you caught (omitting the auth token from my verifying diagnostic attempt to connect to the http endpoint via URL as I have done with all other UN-authenticated endpoints I have ever debugged) SHOULD still have worked I my scraper as it DID have the token but also returns no data if I browse to it in Prometeus.

  - job_name: 'Watchtower'
    scrape_interval: 5m
    scrape_timeout: 1m
    metrics_path: /v1/metrics
    bearer_token: secretTOKEN
    static_configs:
      - targets:
        - '192.168.42.84:9789'

So I am still a bit stumped.... Perhaps you can point me in the right direction? Also on an UNRELATED note can you use the HTTTP API to MANUALLY force a single additional update run, EVEN IF you have it set to update via WATCHTOWER_SCHEDULE ? Documentation leads me to believe not but would be a GREAT additional feature and would also be helpful for diagnostic testing in general (if one manages to get authenticated API access working) :)