[BUG] Grafana can't show metrics which directory names include curly brackets { } if maxBatchSize not 0

[Knud3]

If not 0, carbonapi will do find request to determine how many metrics matches criteria and only then will fetch them, not more than maxBatchSize per request.

Not sure if it is find request or does maxBatchSize do something else for render request.

Maybe related to #29 and/or #746 and/or #661.

Have tried with

unicodeRangeTables:
  - "all"

Tried with escaping curly brackets in directory names Bad Request: Bad Request Target : foo.bar\{topic\}.* Parsed so far : foo.bar\{topic\ Could not parse : }.*

HTTP clients and servers that use path parameters often represent these path parameters as {variable}, e.g. /reports/{date} . It would be great to be able to use this same format in the graphs to refer to these URL endpoints.

Settings in use with newest v0.16.0-patch2 and go-carbon v0.16.2:

listen: "0.0.0.0:8081"
prefix: ""
expvar:
  enabled: false
  pprofEnabled: false
  listen: ""
notFoundStatusCode: 404
unicodeRangeTables:
  - "all"
cache:
  type: "mem"
  size_mb: 1024
  defaultTimeoutSec: 600
backendCache:
  type: "null"
cpus: 0
tz: ""
functionsConfig:
  graphiteWeb: /etc/carbonapi/graphiteWeb.yaml
  timeShift: /etc/carbonapi/timeShift.yaml
graphite:
  host: "carbon-tcp:2003"
  interval: "60s"
  prefix: "carbon.api.gt-grafana-stg"
  pattern: "{prefix}.{fqdn}"
concurency: 5000
idleConnections: 100
pidFile: ""
upstreams:
  graphite09compat: false
  buckets: 10
  concurrencyLimitPerServer: 0
  keepAliveInterval: "10s"
  maxIdleConnsPerHost: 100
  timeouts:
    find: "120s"
    render: "120s"
    connect: "30s"
  backendsv2:
    backends:
      -
        groupName: "go-carbon"
        protocol: "carbonapi_v3_pb"
        lbMethod: "broadcast"
        maxTries: 3
        maxBatchSize: 5000
        concurrencyLimit: 0
        servers:
            - "http://carbonserver:8080"
expireDelaySec: 600
logger:
- logger: ""
  file: "stderr"
  level: "warn"
  encoding: "console"
  encodingTime: "iso8601"
  encodingDuration: "seconds"
- logger: ""
  file: "stdout"
  level: "info"
  encoding: "console"
  encodingTime: "iso8601"
  encodingDuration: "seconds"
- logger: "slow"
  level: "error"

[Knud3]

Tried with different combinations:

filename: test/foo-{0}/total_alloc.wsp

series query: test.*.total_alloc

INFO	access	request served	{"data": {"handler":"find","carbonapi_uuid":"cbfaf034-7681-4e7b-94bc-6622eac34c10","url":"/metrics/find","host":"carbonapi:8081","metrics":["test.*.total_alloc"],"runtime":0.001640143,"http_code":200,"uri":"/metrics/find","from_cache":false,"used_backend_cache":false,"zipper_requests":1,"total_metrics_count":1,"request_headers":{}}}

series query: test.foo-{0}.total_alloc

WARN	zipper	find request failed when resolving globs	{"type": "broadcastGroup", "groupName": "root", "metric_name": "test.foo-{0}.total_alloc", "error": "metric not found"}
INFO	access	request served	{"data": {"handler":"render","carbonapi_uuid":"c4574dd1-f7b3-46f2-b9d3-2f6395aafb03","url":"/render","host":"carbonapi:8081","format":"json","use_cache":true,"targets":["test.foo-{0}.total_alloc"],"cache_timeout":600,"metrics":["test.foo-{0}.total_alloc"],"have_non_fatal_errors":true,"runtime":0.00222413,"http_code":200,"carbonapi_response_size_bytes":2,"from":1676306580,"until":1676328180,"max_data_points":2128,"from_raw":"1676306602","until_raw":"1676328204","uri":"/render","from_cache":false,"used_backend_cache":false,"request_headers":{"X-Panel-Id":"2"}}}

series query: test.foo-\\{0\\}.total_alloc

WARN	zipper	find request failed when resolving globs	{"type": "broadcastGroup", "groupName": "root", "metric_name": "test.foo-\\{0\\}.total_alloc", "error": "metric not found"}

filename: test/foo-\\{0\\}/total_alloc.wsp

series query: test.*.total_alloc

INFO	access	request served	{"data": {"handler":"find","carbonapi_uuid":"73e0486a-0b12-41a7-bed6-590f20b9aa34","url":"/metrics/find","host":"carbonapi:8081","metrics":["test.*.total_alloc"],"runtime":0.001691782,"http_code":200,"uri":"/metrics/find","from_cache":false,"used_backend_cache":false,"zipper_requests":1,"total_metrics_count":1,"request_headers":{}}}
WARN	zipper	find request failed when resolving globs	{"type": "broadcastGroup", "groupName": "go-carbon", "metric_name": "test.foo-\\\\{0\\\\}.total_alloc", "error": "no metrics in the Response"}
ERROR	access	request failed	{"data": {"handler":"render","carbonapi_uuid":"e5019610-f1c7-46e7-904f-780012a120b0","url":"/render","host":"carbonapi:8081","format":"json","use_cache":true,"targets":["test.*.total_alloc"],"cache_timeout":600,"runtime":0.001764256,"http_code":500,"reason":"no metrics in the Response","from":1676305560,"until":1676327160,"from_raw":"1676305601","until_raw":"1676327203","uri":"/render","from_cache":false,"used_backend_cache":false,"request_headers":{"X-Panel-Id":"2"}}}

series query: test.foo-{0}.total_alloc

INFO	access	request served	{"data": {"handler":"render","carbonapi_uuid":"e9c8f3f9-d125-4e86-89fa-41853b10c369","url":"/render","host":"carbonapi:8081","format":"json","use_cache":true,"targets":["test.foo-{0}.total_alloc"],"cache_timeout":600,"metrics":["test.foo-{0}.total_alloc"],"have_non_fatal_errors":true,"runtime":0.002544316,"http_code":200,"carbonapi_response_size_bytes":2,"from":1676305860,"until":1676327460,"max_data_points":2128,"from_raw":"1676305894","until_raw":"1676327496","uri":"/render","from_cache":false,"used_backend_cache":false,"request_headers":{"X-Panel-Id":"2"}}}
INFO	access	request served	{"data": {"handler":"find","carbonapi_uuid":"67f85b0c-0ec4-427f-b97b-de9278489b1d","url":"/metrics/find","host":"carbonapi:8081","metrics":["test.foo-{0}.total_alloc"],"runtime":0.00130966,"http_code":200,"reason":"metric not found","uri":"/metrics/find","from_cache":false,"used_backend_cache":false,"zipper_requests":1,"request_headers":{}}}

series query: test.foo-\\{0\\}.total_alloc

INFO	access	request served	{"data": {"handler":"render","carbonapi_uuid":"2f145320-2189-44a6-b538-ef417c5f6fa1","url":"/render","host":"carbonapi:8081","format":"json","use_cache":true,"targets":["test.foo-\\\\{0\\\\}.total_alloc"],"cache_timeout":600,"metrics":["test.foo-\\\\{0\\\\}.total_alloc"],"have_non_fatal_errors":true,"runtime":0.001477726,"http_code":200,"carbonapi_response_size_bytes":2,"from":1676305980,"until":1676327580,"max_data_points":2128,"from_raw":"1676306022","until_raw":"1676327624","uri":"/render","from_cache":false,"used_backend_cache":false,"request_headers":{"X-Panel-Id":"2"}}}

Also these are from go-carbon

INFO	access	find success	{"handler": "find", "url": "/metrics/find/?format=carbonapi_v3_pb", "carbonzipper_uuid": "2661225a-4a17-43f9-8372-b93d606c7257", "carbonapi_uuid": "2661225a-4a17-43f9-8372-b93d606c7257", "query": ["test.*.total_alloc"], "format": "carbonapi_v3_pb", "runtime_seconds": 0.000056104, "Files": 1, "find_cache_enabled": true, "from_cache": true, "http_code": 200, "lookups": 28}
ERROR	access	find failed	{"handler": "find", "url": "/metrics/find/?format=carbonapi_v3_pb", "carbonapi_uuid": "392061dd-26da-4eb3-a111-0e779d6265b8", "carbonzipper_uuid": "392061dd-26da-4eb3-a111-0e779d6265b8", "query": ["test.foo-{0}.total_alloc"], "format": "carbonapi_v3_pb", "runtime_seconds": 0.000054932, "reason": "Not Found", "error": "Not Found", "http_code": 404}"
ERROR	access	find failed	{"handler": "find", "url": "/metrics/find/?format=carbonapi_v3_pb", "carbonapi_uuid": "2d827c60-60eb-4337-a725-4bced1883725", "carbonzipper_uuid": "2d827c60-60eb-4337-a725-4bced1883725", "query": ["test.foo-\\{0\\}.total_alloc"], "format": "carbonapi_v3_pb", "runtime_seconds": 0.000312228, "reason": "Not Found", "error": "Not Found", "http_code": 404}

[Knud3]

Maybe it is because expandGlobBraces funtion in Go-Carbon carbonserver? https://github.com/go-graphite/go-carbon/blob/master/carbonserver/carbonserver.go#L1434-L1477

[deniszh]

Hi @Knud3,

As far as I remember, curly braces are part of Graphite globs, so they were never allowed characters in metrics, not in go-graphite, neither in python implementation. I'm not sure it it's theoretically possible even - how you distinguish glob from real bracket in filename?

[Civil]

Actually in graphite-web there is a functionality of escaping special characters (and it allows any kind of characters, excluding dots in the name as far as I remember) which is semi-working in carbonapi, but can be in non-working state in go-carbon and graphite-clickhouse (I've never checked to be honest)

[Knud3]

More I've looked into it, more it's started to look like you said. Not sure why, but it is working in render when maxBatchSize is set to 0 and curly brackets are replaced with asterisk in query. e.g.:

filename: test/foo-{0}/total_alloc.wsp
series query: test.foo-*.total_alloc

I'm no Grafana user myself, but trying to provide platform to our devs. So, I was no familiar with globs before this.

That use case comes from our dev.

HTTP clients and servers that use path parameters often represent these path parameters as {variable}, e.g. /reports/{date}

What I know now, I try to shift them to use something else than curly brackets, because I'd like to set maxBatchSize for decreased render times.

Thank you for pointing it out! If you do not see easy workaround, I'm ok to close this issue.

[Civil]

Difference is that with maxBatchSize=0 - carbonapi sends request as-is to backend ("give me metrics").
With maxBatchSize > 0 - it first sends find query and then iterates over result to split one big request into smaller one that are not more than maxBatchSize metrics per request.

Basically that means that backend have a fallback path for /render request that tries to fetch data anyhow, but with /metrics/find it tries to expand it and if that fails - returns nothing.