Skip to content

OSIS Initial Performance Analysis

Jump to bottom
Sai Tharun Thodupunuru edited this page Sep 13, 2021 · 31 revisions

Objective

Checking the performance behavior of the OSIS service on scality cloud. As OSIS is not configured through NGINX, only one node will be configured in vCloudDirector.

  • Tenant Flow: Query/Create/List Tenant

  • User Flow: Query/Get/Create/List User

  • Credential Flow: Query/Get/Create/List Credentials

Test Cases:

  • Base Case:

    • Simple flow → Query/Create/List Tenant, Query/Get/Create/List User, Query/Get/Create/List Credentials

  • Medium load:

    • 1 client (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

    • 2 clients (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

    • 4 clients (threads)

      • (loop count) 100 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 1000 x Tenant Flow, User Flow, Credential Flow

      • (loop count) 10000 x Tenant Flow, User Flow, Credential Flow

  • Ramp Up test - Stress failure test

    • Tenant Flow, User Flow, Credential Flow
Clients (threads) Loop Count(number of times to execute the test per thread) Rampup time: (Ramp-Up Period tells JMeter how long to delay before starting the next user) (in seconds)
128 10 300
256 10 300
512 10 300
128 10 600
256 10 600
512 10 600

Metrics to capture:

  • Elapsed time: Measures the elapsed time from just before sending the request to just after the last chunk of the response has been received,

  • Latency: Measures the latency from just before sending the request to just after the first chunk of the response has been received,

  • Median: Number which divides the samples into two equal halves,

  • 90% Line (90th Percentile): The elapsed time below which 90% of the samples fall

  • 95% Line (95th Percentile): The elapsed time below which 95% of the samples fall

  • Throughput: Calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. The formula is: Throughput = (number of requests) / (total time).

Storage Node metrics using RING grafana dashboards(http:///stats/):

  • Docker Image size

  • RAM/CPU/Disk usage

  • Report numbers to Product for tooling adjustments.

Test Results

1. Base Case:

Simple flow → Query/Create/List Tenant, Query/Get/Create/List User, Query/Get/Create/List Credentials

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 1 1095 1095 1095 1095 1095 1095 1095 0.000% 0.91324
List Tenants 1 204 204 204 204 204 204 204 0.000% 4.90196
Query Tenants 1 205 205 205 205 205 205 205 0.000% 4.87805
Create User 1 1432 1432 1432 1432 1432 1432 1432 0.000% 0.69832
List Users 1 292 292 292 292 292 292 292 0.000% 3.42466
Query Users 1 266 266 266 266 266 266 266 0.000% 3.75940
Get User 1 236 236 236 236 236 236 236 0.000% 4.23729
Create Credential 1 281 281 281 281 281 281 281 0.000% 3.55872
List Credentials 1 319 319 319 319 319 319 319 0.000% 3.13480
Query Credentials 1 278 278 278 278 278 278 278 0.000% 3.59712
Get Credential 1 237 237 237 237 237 237 237 0.000% 4.21941
TOTAL 11 440 278 1095 1095 1432 204 1432 0.000% 2.23532

2. Medium load

i. 1 client (threads)

a.(loop count) 100 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 100 228 214 252 294 385 192 813 0.000% 0.38506
List Tenants 100 222 216 251 265 276 196 279 0.000% 0.38624
Query Tenants 100 203 198 221 228 262 185 296 0.000% 0.38622
Create User 100 386 374 464 494 564 292 565 0.000% 0.38603
List Users 100 210 206 239 244 281 185 307 0.000% 0.38630
Query Users 100 232 228 259 279 284 206 353 0.000% 0.38635
Get User 100 208 203 225 243 282 189 312 0.000% 0.38639
Create Credential 100 246 229 281 295 345 202 1253 0.000% 0.38644
List Credentials 100 227 223 256 269 287 200 294 0.000% 0.38643
Query Credentials 100 218 212 245 262 283 192 291 0.000% 0.38647
Get Credential 100 222 213 253 273 366 195 427 0.000% 0.38650
TOTAL 1100 237 217 304 373 478 185 1253 0.000% 4.19926

b.(loop count) 1000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 1000 216 207 241 259 308 185 1446 0.000% 1.34937
List Tenants 1000 283 263 375 440 599 195 874 0.000% 1.35041
Query Tenants 1000 237 224 292 331 439 185 612 0.000% 1.35033
Create User 1000 299 283 352 378 490 240 1719 0.000% 0.98528
List Users 1000 299 292 365 385 489 195 922 0.000% 0.98535
Query Users 1000 207 202 229 241 266 182 563 0.000% 0.98540
Get User 1000 205 201 226 240 261 181 422 0.000% 0.98575
Create Credential 1000 229 221 254 272 310 196 1549 0.000% 0.49581
List Credentials 1000 1209 1226 1896 2125 2515 220 3220 0.000% 0.49534
Query Credentials 1000 289 288 350 373 425 193 493 0.000% 0.49531
Get Credential 1000 287 285 350 375 436 191 576 0.000% 0.49528
TOTAL 11000 342 247 410 1138 1856 181 3220 0.000% 2.91225

c.(loop count) 10000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput(per second)
Create Tenant 10000 259 247 301 328 425 193 1621 0.000% 0.27638
List Tenants 10000 1728 1831 2409 2639 3204 205 5846 0.000% 0.27635
Query Tenants 10000 1628 1725 2289 2516 3127 191 7090 47.775% 0.27633
Create User 10000 470 443 620 701 915 261 3389 0.000% 0.66208
List Users 10000 562 547 740 821 1013 198 1719 0.000% 0.66208
Get User 10000 234 225 279 305 374 185 741 0.000% 0.66208
Query Users 10000 237 227 278 304 392 186 2232 0.000% 0.66208
Create Credential 10000 256 246 299 323 401 201 1750 0.000% 0.44611
List Credentials 10000 1055 1044 1341 1456 1722 210 3093 0.000% 0.44610
Get Credential 10000 466 459 578 624 745 203 1406 67.050% 0.44609
Query Credentials 10000 458 449 568 618 731 204 1258 67.050% 0.44609
TOTAL 109998 668 427 1736 2019 2555 185 7090 16.535% 1.49231

ii. 2 clients (threads)

a.(loop count) 100 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 200 301 276 378 429 917 204 981 0.000% 1.60905
List Tenants 200 526 431 821 979 1262 293 2424 0.000% 1.61760
Query Tenants 200 398 337 597 696 940 238 1966 0.000% 1.62718
Create User 200 416 378 511 604 1087 279 2408 0.000% 1.65669
List Users 200 292 280 367 411 511 208 678 0.000% 1.65842
Get User 200 247 238 290 315 375 191 417 0.000% 1.65784
Query Users 200 253 242 290 329 457 198 548 0.000% 1.65813
Create Credential 200 273 265 330 346 373 215 387 0.000% 1.68162
List Credentials 200 370 360 488 552 615 225 806 0.000% 1.67942
Get Credential 200 273 264 317 342 388 205 429 0.000% 1.67997
Query Credentials 200 277 270 329 356 398 217 439 0.000% 1.67981
TOTAL 2200 329 289 450 557 920 2 2424 0.000% 6.01130

b.(loop count) 1000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 2000 244 236 281 299 364 193 1905 0.000% 1.70639
List Tenants 2000 508 386 908 990 1218 197 1633 0.000% 1.70511
Query Tenants 2000 414 288 788 865 1054 194 1888 0.000% 1.70387
Create User 2000 348 330 427 465 582 251 1873 0.000% 1.65889
List Users 2000 392 398 511 550 643 211 781 0.000% 1.66055
Get User 2000 226 221 257 278 331 185 403 0.000% 1.66064
Query Users 2000 232 224 265 284 339 184 1199 0.000% 1.66073
Create Credential 2000 263 250 314 343 434 203 1562 0.000% 1.10789
List Credentials 2000 817 856 1146 1248 1490 219 2914 0.000% 1.10755
Get Credential 2000 360 363 448 477 560 202 834 14.250% 1.10753
Query Credentials 2000 359 362 449 476 550 196 661 14.250% 1.10747
TOTAL 22000 378 303 728 909 1171 10 2914 2.592% 5.25415

c.(loop count) 10000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 20000 213 207 230 241 280 182 1524 0.000% 0.82181
List Tenants 20000 1136 1160 1404 1509 1791 191 3224 0.000% 0.82182
Query Tenants 20000 1081 1102 1346 1449 1743 183 2847 66.820% 0.82180
Create User 20000 341 332 408 441 531 230 1772 0.000% 1.74725
List Users 20000 392 392 462 494 577 188 850 0.000% 1.74738
Query Users 20000 202 197 218 227 260 178 1407 0.000% 1.74739
Get User 20000 203 198 226 241 275 177 602 0.000% 1.74750
Create Credential 20000 235 221 276 307 441 189 1756 0.000% 1.30582
List Credentials 20000 798 780 974 1068 1330 204 2902 0.000% 1.30579
Query Credentials 20000 418 408 508 559 705 191 1420 80.170% 1.30579
Get Credential 20000 416 406 505 554 707 192 1598 80.170% 1.30579
TOTAL 220000 499 345 1136 1246 1485 177 3224 14.413% 4.99129

iii. 4 clients (threads)

a.(loop count) 100 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 400 402 342 505 568 1004 225 3991 0.000% 3.39121
List Tenants 400 412 384 563 620 783 235 899 0.000% 3.49889
Query Tenants 400 355 328 486 585 741 227 902 0.000% 3.50508
Create User 400 516 471 666 755 1100 306 2521 0.000% 2.72088
List Users 400 367 351 479 544 672 229 749 0.000% 2.75881
Get User 400 287 274 367 408 469 194 583 0.000% 2.75993
Query Users 400 298 280 376 427 605 206 1001 0.000% 2.76098
Create Credential 400 351 325 429 510 669 231 1575 0.000% 2.40310
List Credentials 400 603 567 859 999 1468 252 1837 0.000% 2.39943
Get Credential 400 349 339 434 488 563 228 602 0.000% 2.39968
Query Credentials 400 354 344 442 473 570 241 658 0.000% 2.39866
TOTAL 4400 390 348 562 658 945 18 3991 0.000% 10.13953

b.(loop count) 1000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 4000 359 310 465 554 819 202 50642 0.000% 0.81913
List Tenants 4000 2291 2363 4231 4805 6688 229 10859 0.000% 0.81932
Query Tenants 4000 2191 2234 4073 4685 6242 207 53567 15.704% 0.81898
Create User 4000 733 634 1029 1230 2008 320 50962 0.000% 0.73567
List Users 4000 783 701 1113 1321 1912 243 50933 0.000% 0.73601
Get User 4000 325 285 413 483 693 198 50465 0.000% 0.73603
Query Users 4000 3554 3241 5027 5760 9205 618 21115 100.000% 0.73593
Create Credential 4000 410 369 580 693 1057 218 3079 0.000% 1.13273
List Credentials 4000 1761 1739 2557 2853 3480 260 6089 0.000% 1.13248
Get Credential 4000 668 590 994 1240 2070 209 3759 41.325% 1.13250
Query Credentials 4000 670 594 1004 1228 1845 211 4215 41.325% 1.13249
TOTAL 44000 1249 632 3224 3922 5496 16 53567 18.032% 3.17494

c.(loop count) 10000 x Tenant Flow, User Flow, Credential Flow

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 40000 457 348 511 581 827 196 51911 0.280% 0.59384
List Tenants 39888 3354 2251 3238 3668 52728 232 105708 0.278% 0.59220
Query Tenants 39777 2857 2153 3105 3475 52149 212 102621 80.197% 0.59054
Create User 40000 736 558 877 1031 1609 273 52377 0.090% 0.92399
List Users 39964 801 623 902 1023 1567 225 51783 0.000% 0.92323
Get User 39964 653 256 363 418 1286 185 100585 0.000% 0.92323
Query Users 39964 2083 1550 2303 2624 4678 303 101982 100.000% 0.92321
Create Credential 40000 299 280 386 426 543 197 1840 0.000% 1.25570
List Credentials 40000 1769 1202 3538 3791 4323 240 26304 0.000% 1.25562
Get Credential 40000 545 528 698 753 892 216 4822 88.335% 1.25561
Query Credentials 40000 560 547 711 768 908 212 5241 88.335% 1.25561
TOTAL 439557 1282 604 2430 3013 4012 9 105708 32.485% 3.08440
Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 40000 299 244 310 350 519 190 50661 0.055% 0.68171
List Tenants 39978 2928 2125 3109 3632 52213 212 102299 0.155% 0.68137
Query Tenants 39916 2598 2099 3027 3498 6046 198 102378 79.860% 0.68030
Create User 40000 708 608 908 1047 1548 279 51864 0.042% 0.93121
List Users 39983 753 659 940 1064 1467 208 51867 0.000% 0.93084
Get User 39983 397 248 340 388 574 186 50719 0.000% 0.93085
Query Users 39983 2372 2230 3241 3742 5124 327 54688 100.000% 0.93084
Create Credential 40000 0 0 0 0 0 0 1 100.000% 1916.71858
List Credentials 40000 0 0 0 0 0 0 1 100.000% 1917.26981
Get Credential 40000 0 0 0 0 0 0 1 100.000% 1917.36171
Query Credentials 40000 0 0 0 0 0 0 1 100.000% 1917.45362
TOTAL 439847 913 276 2352 2766 3921 0 102378 52.737% 4.32683

3. Stress/Rampup testing:

Ramp up Time: 300 seconds

a.Thread count: 128; Loop count: 10; Ramp up Time: 300 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 1280 3690 2906 7264 9710 15256 399 31784 0.000% 2.11747
List Tenants 1280 11134 11668 19757 23017 28701 525 45297 0.000% 2.11803
Query Tenants 1280 10960 11288 19087 22585 29978 378 45998 0.000% 2.11846
Create User 1280 2339 2113 4184 4881 6107 426 7471 0.000% 3.37427
List Users 1280 1921 1841 3396 3930 5310 233 8846 0.000% 3.39433
Get User 1280 1083 1029 1851 2089 2488 245 3371 0.000% 3.39542
Query Users 1280 1140 1033 1934 2254 3084 219 9406 0.000% 3.39581
Create Credential 1280 887 814 1347 1575 2181 273 3762 0.000% 2.06806
List Credentials 1280 23265 21537 42782 46878 49896 294 52313 0.000% 2.02162
Get Credential 1280 1133 1030 1791 2116 2663 277 3883 2.734% 2.02142
Query Credentials 1280 1126 1026 1794 2033 2847 272 4478 2.734% 2.02101
TOTAL 14080 5239 1618 15808 23076 42481 0 52313 0.488% 8.83323

b.Thread count: 256; Loop count: 10; Ramp up Time: 300 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 2560 15427 12157 32612 41460 59611 655 128702 51.523% 3.85725
List Tenants 1241 15158 12059 29485 42617 61206 565 118640 0.242% 1.86439
Query Tenants 1238 17932 12576 44054 51822 70235 588 127242 0.485% 1.85876
Create User 2560 24508 16201 50676 63838 103173 1110 128811 22.773% 3.61519
List Users 1977 13686 6533 36291 41553 116029 547 146557 1.366% 3.13370
Get User 1950 7129 3700 23240 33821 43253 241 180094 0.513% 3.09655
Query Users 1940 5400 3236 7953 27136 40187 251 180099 0.206% 3.09455
Create Credential 2560 17538 6002 42762 54896 82505 341 130912 22.422% 1.62372
List Credentials 2560 43523 40955 79488 88908 102753 367 128065 0.742% 1.61626
Get Credential 2560 32740 38320 57646 68793 115036 318 176244 44.414% 1.61584
Query Credentials 2560 10913 4996 30916 45359 61223 322 119967 43.711% 1.61550
TOTAL 23706 19116 9191 48694 58728 96013 0 180099 19.832% 8.16206

c.Thread count: 512; Loop count: 10; Ramp up Time: 300 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 5120 46604 33756 127227 127264 129551 465 191513 62.676% 4.23159
List Tenants 1911 27507 21774 52969 67506 127282 764 181099 4.448% 1.56582
Query Tenants 1826 40920 29691 115579 127254 132388 433 193898 16.758% 1.49066
Create User 5120 74723 63076 127256 127320 139186 850 191927 47.656% 3.35501
List Users 2680 27537 3708 122552 127266 127344 338 188767 12.948% 1.75822
Get User 2333 9654 1372 36450 67233 127287 279 157355 4.072% 1.53065
Query Users 2238 47275 47400 73409 102288 127320 3989 201481 100.000% 1.46487
Create Credential 5120 26226 4106 114431 127231 128539 349 190400 11.055% 2.26984
List Credentials 5120 45767 37595 93829 127230 137075 476 212294 7.344% 2.26739
Get Credential 5120 62269 47196 127268 127321 147630 336 191381 65.039% 2.26712
Query Credentials 5120 34956 7568 127223 127232 138756 377 190482 61.348% 2.26702
TOTAL 42708 42539 34759 127218 127259 133982 0 212294 37.770% 8.50567

Ramp up Time: 600 seconds

a.Thread count: 128; Loop count: 10; Ramp up Time: 600 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 1280 248 233 310 343 423 170 2602 0.000% 2.12017
List Tenants 1280 356 321 508 642 970 179 1264 0.000% 2.12829
Query Tenants 1280 287 254 384 533 798 164 1124 0.000% 2.12851
Create User 1280 387 372 486 540 650 240 1531 0.000% 2.10583
List Users 1280 354 341 496 548 633 183 887 0.000% 2.10972
Get User 1280 233 226 289 319 373 167 467 0.000% 2.10954
Query Users 1280 238 228 295 321 399 163 1302 0.000% 2.10953
Create Credential 1280 414 344 674 795 1165 195 1599 0.000% 1.95097
List Credentials 1280 3093 1411 7207 7872 9177 199 11146 0.000% 1.94766
Get Credential 1280 499 407 848 965 1280 194 1806 2.812% 1.94792
Query Credentials 1280 490 418 820 936 1204 194 1824 2.812% 1.94773
TOTAL 14080 600 309 767 1254 7036 163 11146 0.511% 7.52315

b.Thread count: 256; Loop count: 10; Ramp up Time: 600 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 2560 1255 465 3396 5431 9923 196 19242 0.000% 3.08291
List Tenants 2560 5324 1562 13969 15973 20674 208 32834 0.000% 3.08840
Query Tenants 2560 5192 1263 13870 15757 20912 193 32122 2.617% 3.08823
Create User 2560 554 532 766 844 987 265 1659 0.000% 4.19142
List Users 2560 457 443 634 713 873 178 1275 0.000% 4.19930
Get User 2560 280 267 367 412 494 167 823 0.000% 4.19946
Query Users 2560 293 273 383 433 544 167 2456 0.000% 4.19961
Create Credential 2560 10781 3450 27987 29324 35753 190 50008 29.922% 2.59994
List Credentials 2560 8996 7924 16482 19795 28927 188 47764 0.156% 2.59663
Get Credential 2560 3621 2871 6751 8831 24780 185 43164 39.844% 2.59548
Query Credentials 2560 3682 3044 6809 9105 20895 179 42979 39.961% 2.59489
TOTAL 28160 3676 617 11447 16472 28299 167 50008 10.227% 11.58087

c.Thread count: 512; Loop count: 10; Ramp up Time: 600 seconds (Tenant Flow, User Flow, Credential Flow)

Service # Samples Average (ms) Median (ms) 90% Line (ms) 95% Line (ms) 99% Line (ms) Min (ms) Max (ms) Error % Throughput (per second)
Create Tenant 5120 21371 14665 46080 101378 127270 209 181110 43.203% 5.05480
List Tenants 2908 13053 11529 24228 32608 67163 223 148948 0.653% 2.86920
Query Tenants 2889 17441 12354 45181 56682 127219 230 157850 12.150% 2.84833
Create User 5120 4244 3099 5327 17935 28083 298 37077 0.000% 6.81191
List Users 5120 4123 3768 5644 6371 26047 217 34906 0.000% 6.82334
Get User 5120 3381 2818 4407 5146 26817 198 30863 0.000% 6.82406
Query Users 5120 3349 2821 4383 5341 26384 195 31138 0.000% 6.82435
Create Credential 5120 17959 4377 48224 94351 127260 222 176967 6.465% 3.42256
List Credentials 5120 18313 9449 45046 63286 127231 212 185674 3.477% 3.41997
Get Credential 5120 18020 7755 46955 77388 127254 206 174998 56.641% 3.41971
Query Credentials 5120 17032 7387 46638 75964 127244 212 173996 56.992% 3.41876
TOTAL 51877 12341 4261 33652 49006 127207 195 185674 17.173% 15.88066
Clone this wiki locally