You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a requirement to keep track of some business number over time. The number can go up or down depending on some actions in my API, and I want to send a notification email when it goes below a certain threshold.
To model this, I created counters to keep track of said number and pushed increment/decrement deltas to those counters in my code: this is for a reservation system, so:
when the API's "creates new slots" action is called, the counter is incremented by how many slots were added
when the API's "reserve slots" action is called, I decrement the counter by the amount of slots that were reserved
Due to limitations in the .NET ecosystem, I'm using 2 standard counters for this, instead of using a single up_down_counter:
one counter that I only increment when "slots are added"
a separate counter that I only increment when "slots are reserved"
This gives me 2 monotonically increasing counters that I can then manipulate in my monitoring tool. I can do (added) - (reserved) to get the net total of "free slots". This is a workaround I found for the lack of support for up-down-counters in .NET, which is coming as part of .NET7 in a few months.
However, standard counters are normally mapped to "relative" counters in most monitoring applications, including Datadog. If I want to know "how many slots were added", I have to sum the counter over a period of time, as each reading only contains a delta over a small timeframe.
Now, the issue is that I need this counter to represent ALL additions and reservations, since "the beginning of time". I expect that they would provide 100% accurate results for basically "an infinite amount of time", so that I can always get the exact number of slots added and reserved, at any given point, by "summing the counters from the beginning".
After reading the docs on metrics, in particular the concept of "resets" and "start time" however, I got very confused:
The documentation mentions the concept of a "reset", were if the source of the telemetry restarts/goes offline/etc, the next time it starts again, a "new collection cycle" begins. When I first considered counters, I was unaware of this and thought each increment I sent was always in relation to the previous collection, all the way back to the very first collection, regardless of whether or not my API was stopped/restarted/etc. The docs seem to go against that notion though.
Now my question is as follows: how do I "keep track of a long-running counter that should survive restarts"? I need to have 100% accurate readings regarding my 2 counters, and I don't want to convert them to a gauge and have to keep the running total myself. I want my API to just push the deltas and then compute the accumulated value over the entire history ("since the beginning") so that it takes "all API requests ever made" into consideration: if someone did a PUSH /myApi/slots 5 years ago, that value should not be "lost" when computing the current number of added slots over time, otherwise it would result in an incorrect total.
All examples I've seen thus far of custom counters either seem to not be "long-lasting" such as this, or they just completely omit the issue of "how to keep it in sync with the data and get the lifetime sum".
Now I'm confused even with a simpler example... imagine a marketplace where I want to keep track of "how many items have been sold since our company was founded", while pushing a counter increment every time one particular purchase is made. Is a counter not a good fit for this type of "long-running" requirement, or am I missing something obvious?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I have a requirement to keep track of some business number over time. The number can go up or down depending on some actions in my API, and I want to send a notification email when it goes below a certain threshold.
To model this, I created counters to keep track of said number and pushed increment/decrement deltas to those counters in my code: this is for a reservation system, so:
Due to limitations in the .NET ecosystem, I'm using 2 standard counters for this, instead of using a single up_down_counter:
This gives me 2 monotonically increasing counters that I can then manipulate in my monitoring tool. I can do
(added) - (reserved)
to get the net total of "free slots". This is a workaround I found for the lack of support for up-down-counters in .NET, which is coming as part of .NET7 in a few months.However, standard counters are normally mapped to "relative" counters in most monitoring applications, including Datadog. If I want to know "how many slots were added", I have to sum the counter over a period of time, as each reading only contains a delta over a small timeframe.
Now, the issue is that I need this counter to represent ALL additions and reservations, since "the beginning of time". I expect that they would provide 100% accurate results for basically "an infinite amount of time", so that I can always get the exact number of slots added and reserved, at any given point, by "summing the counters from the beginning".
After reading the docs on metrics, in particular the concept of "resets" and "start time" however, I got very confused:
The documentation mentions the concept of a "reset", were if the source of the telemetry restarts/goes offline/etc, the next time it starts again, a "new collection cycle" begins. When I first considered counters, I was unaware of this and thought each increment I sent was always in relation to the previous collection, all the way back to the very first collection, regardless of whether or not my API was stopped/restarted/etc. The docs seem to go against that notion though.
Now my question is as follows: how do I "keep track of a long-running counter that should survive restarts"? I need to have 100% accurate readings regarding my 2 counters, and I don't want to convert them to a gauge and have to keep the running total myself. I want my API to just push the deltas and then compute the accumulated value over the entire history ("since the beginning") so that it takes "all API requests ever made" into consideration: if someone did a
PUSH /myApi/slots
5 years ago, that value should not be "lost" when computing the current number of added slots over time, otherwise it would result in an incorrect total.All examples I've seen thus far of custom counters either seem to not be "long-lasting" such as this, or they just completely omit the issue of "how to keep it in sync with the data and get the lifetime sum".
Now I'm confused even with a simpler example... imagine a marketplace where I want to keep track of "how many items have been sold since our company was founded", while pushing a counter increment every time one particular purchase is made. Is a counter not a good fit for this type of "long-running" requirement, or am I missing something obvious?
Beta Was this translation helpful? Give feedback.
All reactions