how can i convert log to gauge ?

[jluncc]

Component(s)

processor/transform

Is your feature request related to a problem? Please describe.

hi, i have some request log, like

request_url1,200,2  
request_url2,500,1

this means request_url_name, response_code, time

now i can receive this log on the opentelemetry collector receiver, the next step i want to convert this log to gauge metric (just like request_url_info{url="request_url1",code="200"} 2 ), and then expose it let prometheus to consume so that we can query on the guafana.

my question is how can i convert the log to gauge metric ?

--

i have search seems if i want to convert log to metric, i need connector, and i find one: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/signaltometricsconnector, but this can not convert to gauge,

so is there any ideas can help me to do that ?

Describe the solution you'd like

maybe some processor or connector can help me to do that ?

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jaronoff97]

@jluncc for your use case it seems that you want to keep track of http responses which is usually better expressed as a counter. Can you explain your use case a bit more for a gauge?

[jluncc]

@jluncc for your use case it seems that you want to keep track of http responses which is usually better expressed as a counter. Can you explain your use case a bit more for a gauge?

@jaronoff97 thanks for your reply!

for example my log is

request_url1, ip1, env1, content-type, response-code, 2  

and i want to convert to gauge like

request_url_info{url="request_url1", ip="ip1", env="env1", content-type="content-type", code="response-code"} 2

the value is response time,

and then expose to prometheus, use promql to compuate reqeust info with label filter or aggregate function flexible

---

however now i use fluentbit to convert log to gauge: https://docs.fluentbit.io/manual/pipeline/filters/log_to_metrics#configuration-gauge

but i want to know is i can do the same thing on collector ?

[jaronoff97]

@jluncc response time is usually better expressed as a histogram (per the semantic convention). Converting it to a histogram will allow you to keep and understand the whole picture of your response times whereas a gauge will have some amount of loss when rolling up over time windows.

[jluncc]

@jluncc response time is usually better expressed as a histogram (per the semantic convention). Converting it to a histogram will allow you to keep and understand the whole picture of your response times whereas a gauge will have some amount of loss when rolling up over time windows.

@jaronoff97 seems on OpenTelemetry Collector now, we have no way to convert log to histogram?

i find this connector show me can do that: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/signaltometricsconnector#histogram, but will throw no exist error when i use it.

and i still wonder if we can convert log to gauge ? above signaltometricsconnector just show me how to convert log to Sum/Histogram/Exponential Histogram, not include gauge

[jaronoff97]

What error do you get when trying to convert to a histogram? Can you share your configuration too?

[jluncc]

What error do you get when trying to convert to a histogram? Can you share your configuration too?

docker image version: opentelemetry-collector-contrib:0.114.0

error log:

Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
error decoding 'connectors': unknown type: "signaltometrics" for id: "signaltometrics" (valid values: [datadog grafanacloud roundrobin servicegraph spanmetrics forward count exceptions otlpjson routing])

my config:

    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 3300
        spike_limit_mib: 650
      batch:
        send_batch_size: 1500
        send_batch_max_size: 3000
        timeout: 2s    
    connectors:
      signaltometrics:
        logs:
          - name: test.histogram
            description: test histogram compuate
            histogram:
              value: Int(attributes["request_time"])
    exporters:
      debug:
        verbosity: normal         
    service:    
      pipelines:
        logs:
          receivers: [otlp]
          processors: [memory_limiter,batch]
          exporters: [debug,signaltometrics]                  
        metrics:
          receivers: [signaltometrics]
          processors: [memory_limiter,batch]
          exporters: [debug]

[jluncc]

oh, just see 0.115 version include this connector component, so 0.114 not work

https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.115.0

but i still want to know is there a way to convert log to gauge ?

[lahsivjar]

The signaltometrics component is still in development phase, so, you will not be able to use it yet. If you are willing to build your own collector using ocb, then you can use the signaltometrics connector from here.

but i still want to know is there a way to convert log to gauge ?

Unfortunately, the component doesn't support gauges. That being said, histogram is a better way to expose these metrics but you can also use sum metric type.

[jluncc]

The signaltometrics component is still in development phase, so, you will not be able to use it yet. If you are willing to build your own collector using ocb, then you can use the signaltometrics connector from here.

but i still want to know is there a way to convert log to gauge ?

Unfortunately, the component doesn't support gauges. That being said, histogram is a better way to expose these metrics but you can also use sum metric type.

@lahsivjar noted, thanks your reply again

[howan36]

@jluncc , hello you could use this connector in version 115.0?

[jluncc]

@jluncc , hello you could use this connector in version 115.0?

@howan36 sry, not to try, but as mentioned above, signaltometricsconnector still under development phase, so i think not recommended now

[howan36]

@jluncc , hello you could use this connector in version 115.0?

@howan36 sry, not to try, but as mentioned above, signaltometricsconnector still under development phase, so i think not recommended now

i have tried with 116.1, still not worked, maybe need to build by self as mentioned.

[howan36]

The signaltometrics component is still in development phase, so, you will not be able to use it yet. If you are willing to build your own collector using ocb, then you can use the signaltometrics connector from here.

but i still want to know is there a way to convert log to gauge ?

Unfortunately, the component doesn't support gauges. That being said, histogram is a better way to expose these metrics but you can also use sum metric type.

i saw the version 116.0 of otel collector contrib has included the signaltometrics connector, if i still need to build it then could be able to use that?

[lahsivjar]

👋 @howan36 , the component is still in development phase, it can only be used once it is promoted from that stage (you can check out the definitions of the stability levels here). To give you a better timeline, once this PR is merged the component should be usable and it will be promoted to beta post that.

[howan36]

👋 @howan36 , the component is still in development phase, it can only be used once it is promoted from that stage (you can check out the definitions of the stability levels here). To give you a better timeline, once this PR is merged the component should be usable and it will be promoted to beta post that.

@lahsivjar thanks for that information, i built it for customized collector,and i saw it has been merged, one more question, for histogram type metrics, i saw it will automatically carry the value which is used to calculate the bucket as label, that means i need to manually remove that by processor, or others?

[howan36]

as my understanding, histogram type metrics should not carry that label,because it will increase the cardinality of the metrics, and that value of label is not useful

[lahsivjar]

i saw it will automatically carry the value which is used to calculate the bucket as label, that means i need to manually remove that by processor, or others?

Sorry, I didn't get this. Can you share an example?

[howan36]

i saw it will automatically carry the value which is used to calculate the bucket as label, that means i need to manually remove that by processor, or others?

Sorry, I didn't get this. Can you share an example?

my config histogram:
count: "1"
value: attributes["request.time"]
buckets: [1, 2, 4, 8, 10, 20, 40, 50, 100, 200,500,1000,2000,3000]
the generated metric:
dcdn_request_time_histogram_bucket{domain="srv.xxx.cn",uri="/accounts/mexx",request_time="1234"}

i mean the labels of request_time should not kept

[lahsivjar]

Did you also configure anything in the attributes config? By default the component will not add any attributes to the generated metric if attributes is not configured (ref) so the behaviour might be an edge-case or bug if it is not a misconfiguration.

Also, is dcdn_request_time_histogram_bucket the name of the configured metric? Can you share the full signaltometrics configuration you are using?

[howan36]

Did you also configure anything in the attributes config? By default the component will not add any attributes to the generated metric if attributes is not configured (ref) so the behaviour might be an edge-case or bug if it is not a misconfiguration.

Also, is dcdn_request_time_histogram_bucket the name of the configured metric? Can you share the full signaltometrics configuration you are using?

connectors:
signaltometrics:
logs:
- name: dcdn.request.time.histogram
description: Histogram for the request time field of dcdn logs
conditions: # Will evaluate to true
- resource.attributes["service.name"] == "dcdn"
# attributes:
# - key: request.time
histogram:
count: "1"
value: attributes["request.time"]
buckets: [1, 2, 4, 8, 10, 20, 40, 50, 100, 200,500,1000,2000,3000]

[lahsivjar]

@howan36

# attributes:
# - key: request.time

If this section of the config is commented out then you should not see the request.time attribute in the output.

dcdn_request_time_histogram_bucket{domain="srv.xxx.cn",uri="/accounts/mexx",request_time="1234"}

I cannot reproduce this in my local environment. I also have a test here which essentially does the same thing and if you see the test output, it doesn't have any log record attributes.

If you are still seeing the attribute in the output even though you don't have attributes configured, please open a bug issue with the full otel config and debug output logs.