Skip to content

Latest commit

 

History

History
165 lines (154 loc) · 25.7 KB

ObservedRemoveSet.md

File metadata and controls

165 lines (154 loc) · 25.7 KB

Results of ObservedRemoveSets V2 and V3 benchmarks were completely unexpected to me.

V2 was performing better in almost every metric except removal (all the time) and memory allocations (only with high Count) Here are the raw results:

BenchmarkDotNet=v0.13.1, OS=macOS Monterey 12.4 (21F79) [Darwin 21.5.0]
Apple M1, 1 CPU, 8 logical and 8 physical cores
.NET SDK=6.0.300
[Host]   : .NET 6.0.5 (6.0.522.21309), Arm64 RyuJIT
ShortRun : .NET 6.0.5 (6.0.522.21309), Arm64 RyuJIT

Job=ShortRun  IterationCount=3  LaunchCount=1  WarmupCount=3
Method Count NumberOfActors Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
AddV2 1000 1 725,253.99 ns 1,763,065.939 ns 96,639.593 ns 99.6094 0.9766 0.9766 470,150 B
AddV3 1000 1 1,860,969.32 ns 853,236.123 ns 46,768.751 ns 109.3750 35.1563 - 832,987 B
RemoveV2_Full 1000 1 374.73 ns 10.268 ns 0.563 ns 0.0648 - - 136 B
RemoveV3_Full 1000 1 41.74 ns 16.844 ns 0.923 ns - - - -
RemoveV2_HalfRemoved 1000 1 373.35 ns 35.178 ns 1.928 ns 0.0648 - - 136 B
RemoveV3_HalfRemoved 1000 1 40.36 ns 2.643 ns 0.145 ns - - - -
AddRemoveV2 1000 1 764,112.89 ns 645,522.180 ns 35,383.249 ns 107.4219 1.9531 1.9531 502,426 B
AddRemoveV3 1000 1 1,512,872.79 ns 2,736,056.792 ns 149,972.505 ns 54.6875 11.7188 - 353,533 B
EnumerateDeltasV2_Full 1000 1 22,535.38 ns 4,354.020 ns 238.659 ns 23.1934 - - 48,560 B
EnumerateDeltasV3_Full 1000 1 38,536.64 ns 66,977.820 ns 3,671.280 ns 23.0713 - - 48,328 B
EnumerateDeltasV2_HalfRemoved 1000 1 22,823.52 ns 81,440.945 ns 4,464.053 ns 27.5574 - - 57,688 B
EnumerateDeltasV3_HalfRemoved 1000 1 23,189.41 ns 3,238.756 ns 177.527 ns 17.4255 - - 36,472 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 1000 3 1,166,299.93 ns 810,964.677 ns 44,451.710 ns 108.3984 1.9531 1.9531 716,146 B
AddV3 1000 3 2,297,894.49 ns 2,246,116.293 ns 123,117.213 ns 111.3281 39.0625 - 842,652 B
RemoveV2_Full 1000 3 415.70 ns 25.305 ns 1.387 ns 0.0725 - - 152 B
RemoveV3_Full 1000 3 40.92 ns 7.319 ns 0.401 ns - - - -
RemoveV2_HalfRemoved 1000 3 408.12 ns 17.555 ns 0.962 ns 0.0725 - - 152 B
RemoveV3_HalfRemoved 1000 3 40.13 ns 2.458 ns 0.135 ns - - - -
AddRemoveV2 1000 3 1,049,929.78 ns 554,169.395 ns 30,375.895 ns 113.2813 - - 354,384 B
AddRemoveV3 1000 3 1,460,148.80 ns 3,692,315.955 ns 202,388.297 ns 58.5938 17.5781 - 353,408 B
EnumerateDeltasV2_Full 1000 3 23,212.03 ns 2,417.605 ns 132.517 ns 23.4375 - - 49,040 B
EnumerateDeltasV3_Full 1000 3 34,342.39 ns 8,410.212 ns 460.992 ns 23.1323 - - 48,408 B
EnumerateDeltasV2_HalfRemoved 1000 3 18,269.50 ns 1,503.208 ns 82.396 ns 25.7874 - - 53,944 B
EnumerateDeltasV3_HalfRemoved 1000 3 23,069.59 ns 8,860.415 ns 485.669 ns 17.5171 - - 36,648 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 1000 10 2,519,994.86 ns 1,156,174.942 ns 63,373.850 ns 132.8125 - - 280,002 B
AddV3 1000 10 2,342,268.31 ns 3,881,848.296 ns 212,777.204 ns 109.3750 37.1094 - 840,507 B
RemoveV2_Full 1000 10 670.30 ns 57.262 ns 3.139 ns 0.0992 - - 208 B
RemoveV3_Full 1000 10 40.62 ns 14.238 ns 0.780 ns - - - -
RemoveV2_HalfRemoved 1000 10 692.76 ns 45.706 ns 2.505 ns 0.0992 - - 208 B
RemoveV3_HalfRemoved 1000 10 40.59 ns 0.976 ns 0.054 ns - - - -
AddRemoveV2 1000 10 1,979,499.81 ns 1,588,898.271 ns 87,092.876 ns 138.6719 - - 565,929 B
AddRemoveV3 1000 10 1,608,002.24 ns 888,720.954 ns 48,713.794 ns 58.5938 17.5781 - 240,295 B
EnumerateDeltasV2_Full 1000 10 25,733.53 ns 1,116.988 ns 61.226 ns 24.2310 - - 50,720 B
EnumerateDeltasV3_Full 1000 10 35,161.16 ns 8,479.352 ns 464.782 ns 23.2544 - - 48,688 B
EnumerateDeltasV2_HalfRemoved 1000 10 20,386.80 ns 1,794.959 ns 98.388 ns 25.1770 - - 52,680 B
EnumerateDeltasV3_HalfRemoved 1000 10 23,717.85 ns 6,009.011 ns 329.374 ns 17.5171 - - 36,688 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 10000 1 6,580,554.86 ns 5,019,632.038 ns 275,142.970 ns 992.1875 - - 2,080,006 B
AddV3 10000 1 24,432,973.96 ns 111,807,172.928 ns 6,128,528.425 ns 468.7500 156.2500 - 2,881,277 B
RemoveV2_Full 10000 1 360.53 ns 13.006 ns 0.713 ns 0.0648 - - 136 B
RemoveV3_Full 10000 1 41.12 ns 5.099 ns 0.279 ns - - - -
RemoveV2_HalfRemoved 10000 1 360.89 ns 12.346 ns 0.677 ns 0.0648 - - 136 B
RemoveV3_HalfRemoved 10000 1 40.03 ns 2.666 ns 0.146 ns - - - -
AddRemoveV2 10000 1 7,886,296.88 ns 13,487,223.671 ns 739,280.240 ns 1054.6875 - - 2,212,097 B
AddRemoveV3 10000 1 13,959,839.74 ns 1,843,396.625 ns 101,042.789 ns 421.8750 109.3750 - 2,404,834 B
EnumerateDeltasV2_Full 10000 1 219,765.62 ns 4,406.435 ns 241.532 ns 229.7363 - - 480,560 B
EnumerateDeltasV3_Full 10000 1 369,240.38 ns 20,385.350 ns 1,117.390 ns 229.4922 - - 480,328 B
EnumerateDeltasV2_HalfRemoved 10000 1 169,438.20 ns 13,603.821 ns 745.671 ns 250.0000 - - 532,320 B
EnumerateDeltasV3_HalfRemoved 10000 1 241,335.77 ns 89,256.593 ns 4,892.455 ns 170.8984 - - 358,312 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 10000 3 12,992,801.43 ns 31,020,237.878 ns 1,700,323.911 ns 1062.5000 - - 2,240,013 B
AddV3 10000 3 24,381,472.23 ns 96,659,834.707 ns 5,298,251.704 ns 437.5000 156.2500 - 2,880,882 B
RemoveV2_Full 10000 3 440.73 ns 10.725 ns 0.588 ns 0.0725 - - 152 B
RemoveV3_Full 10000 3 40.15 ns 4.134 ns 0.227 ns - - - -
RemoveV2_HalfRemoved 10000 3 411.58 ns 106.610 ns 5.844 ns 0.0725 - - 152 B
RemoveV3_HalfRemoved 10000 3 40.55 ns 4.016 ns 0.220 ns - - - -
AddRemoveV2 10000 3 10,468,994.68 ns 5,171,754.096 ns 283,481.293 ns 1125.0000 - - 2,372,124 B
AddRemoveV3 10000 3 14,871,713.76 ns 6,619,015.879 ns 362,810.595 ns 562.5000 156.2500 - 2,405,900 B
EnumerateDeltasV2_Full 10000 3 224,687.41 ns 131,418.722 ns 7,203.504 ns 229.9805 - - 481,040 B
EnumerateDeltasV3_Full 10000 3 378,274.69 ns 371,042.743 ns 20,338.105 ns 229.4922 - - 480,408 B
EnumerateDeltasV2_HalfRemoved 10000 3 169,033.57 ns 33,810.510 ns 1,853.268 ns 238.5254 - - 498,896 B
EnumerateDeltasV3_HalfRemoved 10000 3 234,326.44 ns 89,073.211 ns 4,882.403 ns 172.1191 - - 360,216 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 10000 10 22,715,756.97 ns 39,553,765.409 ns 2,168,075.350 ns 1312.5000 - - 5,421,480 B
AddV3 10000 10 25,906,378.26 ns 93,049,904.200 ns 5,100,379.232 ns 468.7500 156.2500 - 2,878,399 B
RemoveV2_Full 10000 10 703.30 ns 35.151 ns 1.927 ns 0.0992 - - 208 B
RemoveV3_Full 10000 10 40.10 ns 4.282 ns 0.235 ns - - - -
RemoveV2_HalfRemoved 10000 10 690.17 ns 41.902 ns 2.297 ns 0.0992 - - 208 B
RemoveV3_HalfRemoved 10000 10 40.51 ns 5.988 ns 0.328 ns - - - -
AddRemoveV2 10000 10 20,076,399.85 ns 22,885,879.217 ns 1,254,452.265 ns 1390.6250 - - 2,932,015 B
AddRemoveV3 10000 10 15,919,299.81 ns 15,900,582.717 ns 871,564.593 ns 578.1250 156.2500 - 2,405,296 B
EnumerateDeltasV2_Full 10000 10 224,435.54 ns 25,931.031 ns 1,421.367 ns 230.7129 - - 482,720 B
EnumerateDeltasV3_Full 10000 10 355,363.37 ns 233,602.300 ns 12,804.530 ns 229.4922 - - 480,688 B
EnumerateDeltasV2_HalfRemoved 10000 10 171,391.08 ns 5,406.724 ns 296.361 ns 242.6758 - - 507,784 B
EnumerateDeltasV3_HalfRemoved 10000 10 222,866.77 ns 94,045.308 ns 5,154.941 ns 174.3164 - - 364,816 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 100000 1 65,332,522.54 ns 50,599,540.705 ns 2,773,531.567 ns 9875.0000 - - 20,800,555 B
AddV3 100000 1 205,716,569.42 ns 92,488,994.247 ns 5,069,633.865 ns 4750.0000 1750.0000 - 28,791,484 B
RemoveV2_Full 100000 1 373.47 ns 95.343 ns 5.226 ns 0.0648 - - 136 B
RemoveV3_Full 100000 1 40.14 ns 1.219 ns 0.067 ns - - - -
RemoveV2_HalfRemoved 100000 1 336.56 ns 28.878 ns 1.583 ns 0.0648 - - 136 B
RemoveV3_HalfRemoved 100000 1 41.16 ns 39.796 ns 2.181 ns - - - -
AddRemoveV2 100000 1 103,969,102.42 ns 571,123,354.717 ns 31,305,198.241 ns 10625.0000 125.0000 125.0000 58,118,220 B
AddRemoveV3 100000 1 162,568,397.20 ns 342,030,068.743 ns 18,747,822.197 ns 4200.0000 1200.0000 - 24,049,824 B
EnumerateDeltasV2_Full 100000 1 2,388,685.74 ns 3,230,922.987 ns 177,097.791 ns 2292.9688 - - 4,800,565 B
EnumerateDeltasV3_Full 100000 1 6,408,102.21 ns 2,256,764.376 ns 123,700.871 ns 2289.0625 - - 4,800,335 B
EnumerateDeltasV2_HalfRemoved 100000 1 2,782,285.21 ns 1,783,389.916 ns 97,753.619 ns 1816.4063 39.0625 39.0625 5,040,912 B
EnumerateDeltasV3_HalfRemoved 100000 1 5,285,377.28 ns 8,953,939.179 ns 490,795.620 ns 1718.7500 - - 3,602,927 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 100000 3 116,997,023.11 ns 214,425,308.928 ns 11,753,374.728 ns 10666.6667 - - 55,954,845 B
AddV3 100000 3 248,998,356.44 ns 1,217,294,515.141 ns 66,724,019.951 ns 4333.3333 1333.3333 - 28,797,179 B
RemoveV2_Full 100000 3 427.35 ns 43.578 ns 2.389 ns 0.0725 - - 152 B
RemoveV3_Full 100000 3 40.51 ns 10.414 ns 0.571 ns - - - -
RemoveV2_HalfRemoved 100000 3 437.08 ns 27.580 ns 1.512 ns 0.0725 - - 152 B
RemoveV3_HalfRemoved 100000 3 40.11 ns 1.708 ns 0.094 ns - - - -
AddRemoveV2 100000 3 107,765,655.11 ns 54,133,088.452 ns 2,967,217.242 ns 11333.3333 - - 43,249,431 B
AddRemoveV3 100000 3 153,691,591.67 ns 63,454,657.933 ns 3,478,163.919 ns 5000.0000 1200.0000 - 24,069,410 B
EnumerateDeltasV2_Full 100000 3 2,227,378.50 ns 720,980.934 ns 39,519.398 ns 2292.9688 - - 4,801,044 B
EnumerateDeltasV3_Full 100000 3 12,081,142.59 ns 6,784,848.199 ns 371,900.424 ns 2281.2500 - - 4,800,421 B
EnumerateDeltasV2_HalfRemoved 100000 3 3,062,748.75 ns 1,033,959.898 ns 56,674.831 ns 1953.1250 46.8750 46.8750 5,578,276 B
EnumerateDeltasV3_HalfRemoved 100000 3 4,246,412.00 ns 2,498,375.925 ns 136,944.415 ns 1718.7500 - - 3,596,862 B
------------------------------- -------- ---------------- ------------------- ---------------------- ------------------- ------------ ----------- ---------- --------------
AddV2 100000 10 288,586,680.56 ns 1,777,022,987.404 ns 97,404,626.235 ns 13333.3333 - - 55,962,459 B
AddV3 100000 10 232,994,680.44 ns 1,064,711,018.474 ns 58,360,403.629 ns 4333.3333 1333.3333 - 28,784,952 B
RemoveV2_Full 100000 10 690.60 ns 53.338 ns 2.924 ns 0.0992 - - 208 B
RemoveV3_Full 100000 10 40.36 ns 4.340 ns 0.238 ns - - - -
RemoveV2_HalfRemoved 100000 10 702.62 ns 76.001 ns 4.166 ns 0.0992 - - 208 B
RemoveV3_HalfRemoved 100000 10 40.17 ns 3.079 ns 0.169 ns - - - -
AddRemoveV2 100000 10 173,480,763.92 ns 265,691,993.615 ns 14,563,474.708 ns 14000.0000 - - 29,322,864 B
AddRemoveV3 100000 10 156,536,405.53 ns 208,492,789.540 ns 11,428,193.322 ns 5000.0000 1200.0000 - 24,065,598 B
EnumerateDeltasV2_Full 100000 10 2,220,573.51 ns 220,111.661 ns 12,065.063 ns 2296.8750 - - 4,802,724 B
EnumerateDeltasV3_Full 100000 10 12,689,432.29 ns 2,545,811.934 ns 139,544.543 ns 2281.2500 - - 4,800,701 B
EnumerateDeltasV2_HalfRemoved 100000 10 1,716,698.51 ns 365,344.575 ns 20,025.769 ns 2500.0000 - - 5,317,632 B
EnumerateDeltasV3_HalfRemoved 100000 10 4,536,464.20 ns 6,939,420.712 ns 380,373.065 ns 1718.7500 - - 3,597,334 B

Attempt to add concurrency showcases a more complicated picture:

BenchmarkDotNet=v0.13.1, OS=macOS Monterey 12.4 (21F79) [Darwin 21.5.0]
Apple M1, 1 CPU, 8 logical and 8 physical cores
.NET SDK=6.0.300
[Host]     : .NET 6.0.5 (6.0.522.21309), Arm64 RyuJIT
Job-YCGTHV : .NET 6.0.5 (6.0.522.21309), Arm64 RyuJIT

IterationCount=10  LaunchCount=1  WarmupCount=3
Method Count Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
AddAndMergeConcurrentlyV2 1000 12.511 ms 0.2954 ms 0.1954 ms 406.2500 - - 858 KB
AddAndMergeConcurrentlyV3 1000 4.748 ms 0.6073 ms 0.3614 ms 320.3125 109.3750 - 927 KB
--------------------------- --------- -------------: --------------: --------------: ------------: -----------: ----------: -----------:
AddAndMergeConcurrentlyV2 10000 34.187 ms 4.2709 ms 2.8249 ms 3066.6667 400.0000 66.6667 8,425 KB
AddAndMergeConcurrentlyV3 10000 67.392 ms 16.6187 ms 10.9922 ms 3500.0000 1500.0000 666.6667 8,594 KB
--------------------------- --------- -------------: --------------: --------------: ------------: -----------: ----------: -----------:
AddAndMergeConcurrentlyV2 50000 220.961 ms 31.0203 ms 20.5180 ms 16000.0000 2000.0000 - 45,858 KB
AddAndMergeConcurrentlyV3 50000 304.505 ms 51.1833 ms 30.4584 ms 12000.0000 5000.0000 1000.0000 40,845 KB
--------------------------- --------- -------------: --------------: --------------: ------------: -----------: ----------: -----------:
AddAndMergeConcurrentlyV2 100000 425.059 ms 55.3642 ms 36.6200 ms 31000.0000 2000.0000 - 100,013 KB
AddAndMergeConcurrentlyV3 100000 668.540 ms 101.9848 ms 67.4566 ms 20000.0000 8000.0000 2000.0000 91,728 KB
--------------------------- --------- -------------: --------------: --------------: ------------: -----------: ----------: -----------:
AddAndMergeConcurrentlyV2 1000000 8,741.451 ms 1,520.5264 ms 1,005.7341 ms 313000.0000 3000.0000 1000.0000 890,994 KB
AddAndMergeConcurrentlyV3 1000000 8,207.853 ms 1,822.8083 ms 1,205.6748 ms 195000.0000 56000.0000 6000.0000 696,770 KB