Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dogstatsd): compress metrics #901

Merged
merged 6 commits into from
Feb 27, 2025
Merged

feat(dogstatsd): compress metrics #901

merged 6 commits into from
Feb 27, 2025

Conversation

astuyve
Copy link
Contributor

@astuyve astuyve commented Feb 26, 2025

  • feat: use zstd for dogstatsd
  • feat: licenses

What does this PR do?

Uses zstd for dogstatsd. Not optional, always enabled.

Motivation

should help a small bit with network bytes:
Screenshot 2025-02-26 at 2 52 29 PM

@astuyve astuyve requested review from a team as code owners February 26, 2025 19:54
@codecov-commenter
Copy link

codecov-commenter commented Feb 26, 2025

Codecov Report

Attention: Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.

Project coverage is 71.77%. Comparing base (a43896a) to head (0c80cbe).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #901      +/-   ##
==========================================
+ Coverage   71.76%   71.77%   +0.01%     
==========================================
  Files         328      328              
  Lines       48702    48712      +10     
==========================================
+ Hits        34949    34962      +13     
+ Misses      13753    13750       -3     
Components Coverage Δ
crashtracker 42.90% <ø> (ø)
crashtracker-ffi 6.25% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 92.21% <ø> (ø)
data-pipeline-ffi 90.39% <ø> (ø)
ddcommon 80.08% <ø> (ø)
ddcommon-ffi 61.05% <ø> (ø)
ddtelemetry 61.74% <ø> (+0.14%) ⬆️
ddtelemetry-ffi 22.46% <ø> (ø)
dogstatsd 90.02% <78.57%> (-0.15%) ⬇️
dogstatsd-client 82.57% <ø> (ø)
ipc 82.63% <ø> (-0.11%) ⬇️
profiling 80.04% <ø> (ø)
profiling-ffi 70.68% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 40.23% <ø> (ø)
sidecar-ffi 0.14% <ø> (ø)
spawn-worker 54.37% <ø> (ø)
tinybytes 92.00% <ø> (ø)
trace-mini-agent 72.48% <ø> (ø)
trace-normalization 98.23% <ø> (ø)
trace-obfuscation 95.96% <ø> (ø)
trace-protobuf 78.13% <ø> (ø)
trace-utils 93.29% <ø> (ø)

@pr-commenter
Copy link

pr-commenter bot commented Feb 26, 2025

Benchmarks

Comparison

Benchmark execution time: 2025-02-27 19:00:14

Comparing candidate commit 0c80cbe in PR branch aj/compress-metrics with baseline commit a43896a in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 3.895µs 3.914µs ± 0.003µs 3.914µs ± 0.001µs 3.915µs 3.918µs 3.920µs 3.922µs 0.19% -2.101 16.822 0.06% 0.000µs 1 200
credit_card/is_card_number/ throughput 254999349.033op/s 255490415.007op/s ± 164645.774op/s 255486985.338op/s ± 67066.044op/s 255554907.340op/s 255739445.845op/s 255886918.652op/s 256755244.547op/s 0.50% 2.129 17.050 0.06% 11642.214op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 82.109µs 82.449µs ± 0.245µs 82.352µs ± 0.121µs 82.580µs 82.940µs 83.171µs 83.761µs 1.71% 1.632 3.929 0.30% 0.017µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 11938660.337op/s 12128748.273op/s ± 35885.640op/s 12142985.053op/s ± 17816.427op/s 12154014.981op/s 12165393.713op/s 12170041.281op/s 12178858.719op/s 0.30% -1.604 3.750 0.30% 2537.498op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 76.380µs 76.989µs ± 0.276µs 76.965µs ± 0.193µs 77.175µs 77.454µs 77.584µs 78.156µs 1.55% 0.557 0.718 0.36% 0.020µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 12794993.496op/s 12989115.122op/s ± 46535.864op/s 12992951.976op/s ± 32611.996op/s 13025014.218op/s 13053336.783op/s 13079881.769op/s 13092466.856op/s 0.77% -0.532 0.647 0.36% 3290.583op/s 1 200
credit_card/is_card_number/37828224631 execution_time 3.898µs 3.914µs ± 0.002µs 3.914µs ± 0.001µs 3.915µs 3.917µs 3.920µs 3.920µs 0.16% -1.402 11.249 0.06% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 255091165.389op/s 255508649.151op/s ± 144783.063op/s 255511536.349op/s ± 61990.044op/s 255562472.248op/s 255736985.233op/s 255835156.858op/s 256511630.569op/s 0.39% 1.421 11.375 0.06% 10237.709op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 72.925µs 73.852µs ± 0.455µs 73.820µs ± 0.340µs 74.177µs 74.607µs 74.771µs 74.928µs 1.50% 0.148 -0.743 0.61% 0.032µs 1 200
credit_card/is_card_number/378282246310005 throughput 13346220.287op/s 13541129.437op/s ± 83270.255op/s 13546504.190op/s ± 62029.727op/s 13600196.186op/s 13670036.828op/s 13698526.020op/s 13712780.765op/s 1.23% -0.126 -0.750 0.61% 5888.096op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 52.001µs 52.250µs ± 0.110µs 52.256µs ± 0.079µs 52.326µs 52.422µs 52.501µs 52.632µs 0.72% 0.358 0.308 0.21% 0.008µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 18999832.960op/s 19138850.030op/s ± 40403.224op/s 19136602.350op/s ± 28854.788op/s 19169455.873op/s 19201214.381op/s 19217516.645op/s 19230461.050op/s 0.49% -0.344 0.281 0.21% 2856.939op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.468µs 6.657µs ± 0.069µs 6.651µs ± 0.043µs 6.700µs 6.780µs 6.827µs 6.843µs 2.88% 0.161 0.305 1.03% 0.005µs 1 200
credit_card/is_card_number/x371413321323331 throughput 146134494.683op/s 150225890.276op/s ± 1548633.149op/s 150349082.120op/s ± 974963.045op/s 151168751.830op/s 152689731.013op/s 154230679.696op/s 154605738.907op/s 2.83% -0.091 0.307 1.03% 109504.900op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 3.896µs 3.914µs ± 0.003µs 3.914µs ± 0.001µs 3.915µs 3.920µs 3.922µs 3.923µs 0.23% -0.670 10.988 0.07% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 254927051.452op/s 255472982.856op/s ± 178685.024op/s 255503232.116op/s ± 57936.082op/s 255552298.779op/s 255625588.926op/s 255847850.336op/s 256684573.639op/s 0.46% 0.697 11.141 0.07% 12634.939op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 63.695µs 63.860µs ± 0.104µs 63.835µs ± 0.046µs 63.890µs 64.052µs 64.244µs 64.329µs 0.77% 1.781 4.172 0.16% 0.007µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 15545187.037op/s 15659270.490op/s ± 25423.520op/s 15665279.475op/s ± 11233.191op/s 15675529.323op/s 15686444.011op/s 15693345.002op/s 15699821.022op/s 0.22% -1.766 4.104 0.16% 1797.714op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 57.444µs 57.547µs ± 0.071µs 57.531µs ± 0.017µs 57.552µs 57.608µs 57.896µs 57.979µs 0.78% 4.253 20.227 0.12% 0.005µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 17247632.688op/s 17377277.056op/s ± 21295.558op/s 17381849.004op/s ± 5069.340op/s 17386427.720op/s 17391252.461op/s 17398583.269op/s 17408298.536op/s 0.15% -4.237 20.109 0.12% 1505.823op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 3.895µs 3.914µs ± 0.002µs 3.914µs ± 0.001µs 3.915µs 3.918µs 3.921µs 3.924µs 0.26% -1.595 17.945 0.06% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 254840498.861op/s 255499213.318op/s ± 162222.848op/s 255507225.327op/s ± 59153.809op/s 255560260.385op/s 255698639.592op/s 255832752.377op/s 256747249.409op/s 0.49% 1.628 18.177 0.06% 11470.888op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 54.585µs 54.713µs ± 0.133µs 54.678µs ± 0.022µs 54.712µs 54.825µs 55.462µs 55.510µs 1.52% 4.578 22.096 0.24% 0.009µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 18014695.857op/s 18277454.018op/s ± 43857.163op/s 18288730.484op/s ± 7483.114op/s 18294628.240op/s 18300855.199op/s 18303657.017op/s 18320036.624op/s 0.17% -4.555 21.903 0.24% 3101.170op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 51.989µs 52.236µs ± 0.113µs 52.236µs ± 0.084µs 52.315µs 52.411µs 52.511µs 52.561µs 0.62% 0.249 -0.332 0.22% 0.008µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 19025373.705op/s 19143828.185op/s ± 41379.277op/s 19143866.381op/s ± 30703.618op/s 19174901.492op/s 19208505.626op/s 19229468.258op/s 19234965.320op/s 0.48% -0.239 -0.341 0.22% 2925.957op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.511µs 6.673µs ± 0.059µs 6.675µs ± 0.041µs 6.717µs 6.757µs 6.796µs 6.806µs 1.96% -0.276 -0.315 0.88% 0.004µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 146935105.978op/s 149875239.360op/s ± 1324470.585op/s 149811530.411op/s ± 925993.222op/s 150736677.813op/s 152406148.350op/s 152726597.282op/s 153584855.735op/s 2.52% 0.318 -0.283 0.88% 93654.213op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [3.914µs; 3.914µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ throughput [255467596.687op/s; 255513233.328op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [82.415µs; 82.483µs] or [-0.041%; +0.041%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [12123774.868op/s; 12133721.677op/s] or [-0.041%; +0.041%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [76.950µs; 77.027µs] or [-0.050%; +0.050%] None None None
credit_card/is_card_number/ 378282246310005 throughput [12982665.699op/s; 12995564.545op/s] or [-0.050%; +0.050%] None None None
credit_card/is_card_number/37828224631 execution_time [3.913µs; 3.914µs] or [-0.008%; +0.008%] None None None
credit_card/is_card_number/37828224631 throughput [255488583.611op/s; 255528714.691op/s] or [-0.008%; +0.008%] None None None
credit_card/is_card_number/378282246310005 execution_time [73.789µs; 73.915µs] or [-0.085%; +0.085%] None None None
credit_card/is_card_number/378282246310005 throughput [13529588.980op/s; 13552669.893op/s] or [-0.085%; +0.085%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [52.235µs; 52.265µs] or [-0.029%; +0.029%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [19133250.531op/s; 19144449.528op/s] or [-0.029%; +0.029%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.648µs; 6.667µs] or [-0.143%; +0.143%] None None None
credit_card/is_card_number/x371413321323331 throughput [150011264.616op/s; 150440515.937op/s] or [-0.143%; +0.143%] None None None
credit_card/is_card_number_no_luhn/ execution_time [3.914µs; 3.915µs] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ throughput [255448218.830op/s; 255497746.882op/s] or [-0.010%; +0.010%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [63.846µs; 63.875µs] or [-0.023%; +0.023%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [15655747.034op/s; 15662793.945op/s] or [-0.023%; +0.023%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [57.537µs; 57.556µs] or [-0.017%; +0.017%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [17374325.697op/s; 17380228.416op/s] or [-0.017%; +0.017%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [3.914µs; 3.914µs] or [-0.009%; +0.009%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [255476730.791op/s; 255521695.844op/s] or [-0.009%; +0.009%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [54.694µs; 54.731µs] or [-0.034%; +0.034%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [18271375.837op/s; 18283532.199op/s] or [-0.033%; +0.033%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [52.221µs; 52.252µs] or [-0.030%; +0.030%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [19138093.415op/s; 19149562.955op/s] or [-0.030%; +0.030%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.665µs; 6.681µs] or [-0.122%; +0.122%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [149691680.475op/s; 150058798.245op/s] or [-0.122%; +0.122%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 208.608µs 209.079µs ± 0.189µs 209.070µs ± 0.093µs 209.160µs 209.326µs 209.555µs 210.601µs 0.73% 2.668 20.029 0.09% 0.013µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 4748308.020op/s 4782896.334op/s ± 4309.619op/s 4783088.911op/s ± 2138.454op/s 4785229.114op/s 4788443.762op/s 4791419.459op/s 4793669.161op/s 0.22% -2.628 19.640 0.09% 304.736op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 18.230µs 18.323µs ± 0.054µs 18.337µs ± 0.036µs 18.360µs 18.393µs 18.436µs 18.458µs 0.66% -0.323 -0.793 0.30% 0.004µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 54176631.297op/s 54577158.779op/s ± 161667.541op/s 54535584.813op/s ± 107420.226op/s 54732120.772op/s 54843981.781op/s 54854838.264op/s 54855039.795op/s 0.59% 0.333 -0.798 0.30% 11431.621op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 10.661µs 10.728µs ± 0.035µs 10.725µs ± 0.023µs 10.751µs 10.784µs 10.800µs 10.880µs 1.45% 0.527 0.580 0.33% 0.002µs 1 200
normalization/normalize_name/normalize_name/good throughput 91908710.613op/s 93213187.888op/s ± 303379.111op/s 93238622.052op/s ± 203092.529op/s 93437875.960op/s 93647571.843op/s 93722037.416op/s 93802200.022op/s 0.60% -0.504 0.503 0.32% 21452.143op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [209.052µs; 209.105µs] or [-0.013%; +0.013%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [4782299.062op/s; 4783493.606op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [18.315µs; 18.330µs] or [-0.041%; +0.041%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [54554753.213op/s; 54599564.345op/s] or [-0.041%; +0.041%] None None None
normalization/normalize_name/normalize_name/good execution_time [10.723µs; 10.733µs] or [-0.045%; +0.045%] None None None
normalization/normalize_name/normalize_name/good throughput [93171142.461op/s; 93255233.315op/s] or [-0.045%; +0.045%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 6.074ms 6.087ms ± 0.007ms 6.086ms ± 0.003ms 6.090ms 6.097ms 6.100ms 6.145ms 0.97% 3.539 24.087 0.12% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [6.086ms; 6.088ms] or [-0.016%; +0.016%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 149.838µs 150.841µs ± 0.411µs 150.753µs ± 0.179µs 150.978µs 151.398µs 152.424µs 153.590µs 1.88% 2.687 12.212 0.27% 0.029µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [150.784µs; 150.898µs] or [-0.038%; +0.038%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.214µs 3.167µs ± 1.485µs 2.966µs ± 0.024µs 2.989µs 3.602µs 13.826µs 16.300µs 449.59% 7.554 58.160 46.76% 0.105µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.961µs; 3.372µs] or [-6.497%; +6.497%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 53.005ms 53.248ms ± 0.160ms 53.212ms ± 0.062ms 53.279ms 53.507ms 53.900ms 54.189ms 1.84% 2.717 9.971 0.30% 0.011ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [53.226ms; 53.270ms] or [-0.042%; +0.042%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 504.749µs 505.912µs ± 1.544µs 505.777µs ± 0.206µs 505.971µs 506.384µs 507.185µs 521.018µs 3.01% 9.067 85.016 0.30% 0.109µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1919320.100op/s 1976646.458op/s ± 5872.363op/s 1977157.026op/s ± 806.693op/s 1978127.895op/s 1979622.165op/s 1980422.392op/s 1981182.945op/s 0.20% -9.023 84.456 0.30% 415.239op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 452.767µs 453.498µs ± 0.766µs 453.407µs ± 0.187µs 453.622µs 453.914µs 454.538µs 461.057µs 1.69% 7.914 71.351 0.17% 0.054µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2168929.927op/s 2205086.422op/s ± 3673.634op/s 2205526.266op/s ± 908.548op/s 2206338.546op/s 2207676.175op/s 2208395.670op/s 2208643.238op/s 0.14% -7.860 70.661 0.17% 259.765op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 175.649µs 176.589µs ± 0.351µs 176.605µs ± 0.199µs 176.796µs 177.075µs 177.319µs 178.413µs 1.02% 0.476 3.398 0.20% 0.025µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5604979.628op/s 5662901.288op/s ± 11246.203op/s 5662349.149op/s ± 6368.673op/s 5668953.054op/s 5681399.558op/s 5689707.126op/s 5693168.226op/s 0.54% -0.446 3.281 0.20% 795.227op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 37.658µs 37.781µs ± 0.069µs 37.771µs ± 0.042µs 37.820µs 37.895µs 38.000µs 38.160µs 1.03% 1.353 4.305 0.18% 0.005µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 26205291.031op/s 26468508.669op/s ± 48120.170op/s 26475246.547op/s ± 29474.222op/s 26501003.556op/s 26529681.585op/s 26545879.769op/s 26554487.338op/s 0.30% -1.329 4.165 0.18% 3402.610op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 48.208µs 48.317µs ± 0.043µs 48.313µs ± 0.026µs 48.341µs 48.387µs 48.455µs 48.534µs 0.46% 1.064 3.233 0.09% 0.003µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 20604025.746op/s 20696632.074op/s ± 18556.447op/s 20698423.615op/s ± 10952.531op/s 20709095.917op/s 20722623.063op/s 20734208.258op/s 20743273.934op/s 0.22% -1.053 3.189 0.09% 1312.139op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [505.698µs; 506.126µs] or [-0.042%; +0.042%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1975832.605op/s; 1977460.311op/s] or [-0.041%; +0.041%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [453.392µs; 453.604µs] or [-0.023%; +0.023%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2204577.292op/s; 2205595.552op/s] or [-0.023%; +0.023%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [176.540µs; 176.637µs] or [-0.028%; +0.028%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5661342.673op/s; 5664459.904op/s] or [-0.028%; +0.028%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [37.771µs; 37.790µs] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [26461839.676op/s; 26475177.662op/s] or [-0.025%; +0.025%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [48.311µs; 48.323µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [20694060.329op/s; 20699203.819op/s] or [-0.012%; +0.012%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 4.922µs 5.031µs ± 0.036µs 5.042µs ± 0.017µs 5.056µs 5.063µs 5.098µs 5.101µs 1.18% -1.021 0.814 0.72% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.026µs; 5.036µs] or [-0.100%; +0.100%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 32.933µs 33.591µs ± 1.085µs 33.097µs ± 0.053µs 33.194µs 35.901µs 35.935µs 37.660µs 13.78% 1.763 1.371 3.22% 0.077µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [33.441µs; 33.742µs] or [-0.448%; +0.448%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.373µs 2.391µs ± 0.014µs 2.388µs ± 0.006µs 2.395µs 2.434µs 2.438µs 2.440µs 2.17% 1.956 3.757 0.60% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.389µs; 2.393µs] or [-0.084%; +0.084%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 68.425µs 68.619µs ± 0.236µs 68.590µs ± 0.059µs 68.652µs 68.764µs 69.126µs 71.627µs 4.43% 10.530 130.226 0.34% 0.017µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [68.586µs; 68.652µs] or [-0.048%; +0.048%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 245.303ns 254.341ns ± 10.128ns 250.306ns ± 2.913ns 255.582ns 280.395ns 284.649ns 287.309ns 14.78% 1.816 2.364 3.97% 0.716ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [252.937ns; 255.744ns] or [-0.552%; +0.552%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 0c80cbe 1740682106 aj/compress-metrics
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 17.610µs 25.468µs ± 11.016µs 17.736µs ± 0.095µs 35.133µs 41.354µs 54.217µs 95.811µs 440.22% 2.029 8.114 43.15% 0.779µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [23.941µs; 26.994µs] or [-5.995%; +5.995%] None None None

Baseline

Omitted due to size.

@@ -25,6 +25,7 @@ tokio = { version = "1.37.0", default-features = false, features = ["macros", "r
tokio-util = { version = "0.7.11", default-features = false }
tracing = { version = "0.1.40", default-features = false }
regex = { version = "1.10.6", default-features = false }
zstd = { version = "0.13.3", default-features = false }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use gzip or zstd for both logs and metrics? To avoid adding 2 libs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use zstd for both

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@morrisonlevi morrisonlevi changed the title aj/compress metrics feat(dogstatsd): compress metrics Feb 27, 2025
Copy link
Contributor

@morrisonlevi morrisonlevi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, now that the content-encoding matches success/failure of compressing.

@morrisonlevi morrisonlevi merged commit 63b5b7a into main Feb 27, 2025
31 checks passed
@morrisonlevi morrisonlevi deleted the aj/compress-metrics branch February 27, 2025 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants