FEMU Performance Numbers

How much performance to expect from FEMU?

First, For the NoSSD mode, it shows the maximum performance that FEMU platform can deliver.

Host and VM setup: Intel I9-7980XE 36-core (HT enabled) CPU at 3.4GHz (after disabling C-states and running host CPUs in performance mode). The FEMU VM is allocated 8 vCPUs with a guest Linux 4.14. FEMU SSD is 1GB. The rest is the same as in run-nossd.sh.

Here, we use FIO v3.3 for the testing, and the FIO configuration is as below.

In summary, raw FEMU could achieve/scale up to 1 million IOPS (and over 4 GB/s bandwidth)!

Details below:

[global]
ioengine=psync
runtime=60
rw=randread
;threads=128
numjobs=1
bs=4k
size=1G
direct=1
time_based=1
randrepeat=0
norandommap=1
thread
group_reporting
per_job_logs=0

[nvme]
filename=/dev/nvme0n1

We show the results under different queue depths: numjobs set to 1, 4, 16, 64, and 256.

QD=1:

Jobs: 1 (f=1): [r(1)][100.0%][r=461MiB/s,w=0KiB/s][r=118k,w=0 IOPS][eta 00m:00s]
nvme: (groupid=0, jobs=1): err= 0: pid=2370: Fri Dec 11 23:52:32 2020
   read: IOPS=118k, BW=462MiB/s (484MB/s)(27.0GiB/60001msec)
    clat (nsec): min=6835, max=199466, avg=7672.05, stdev=495.34
     lat (nsec): min=6965, max=199594, avg=7802.99, stdev=498.90
    clat percentiles (nsec):
     |  1.00th=[ 7456],  5.00th=[ 7520], 10.00th=[ 7520], 20.00th=[ 7520],
     | 30.00th=[ 7584], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 7648],
     | 70.00th=[ 7648], 80.00th=[ 7712], 90.00th=[ 7776], 95.00th=[ 7840],
     | 99.00th=[10688], 99.50th=[11072], 99.90th=[12480], 99.95th=[12736],
     | 99.99th=[14528]
   bw (  KiB/s): min=471080, max=475440, per=100.00%, avg=472590.99, stdev=616.35, samples=119
   iops        : min=117770, max=118860, avg=118147.76, stdev=154.10, samples=119
  lat (usec)   : 10=98.92%, 20=1.08%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=7.83%, sys=92.16%, ctx=23, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=7089097,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=27.0GiB (29.0GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=7073522/0, merge=0/0, ticks=2048/0, in_queue=0, util=0.00%

QD=4:

nvme: (groupid=0, jobs=4): err= 0: pid=2425: Fri Dec 11 23:56:07 2020
   read: IOPS=341k, BW=1333MiB/s (1398MB/s)(78.1GiB/60014msec)
    clat (usec): min=6, max=39283, avg=10.65, stdev=210.27
     lat (usec): min=6, max=39284, avg=10.82, stdev=211.76
    clat percentiles (usec):
     |  1.00th=[    8],  5.00th=[    8], 10.00th=[    8], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    8], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   12], 99.50th=[   13], 99.90th=[   16], 99.95th=[   19],
     | 99.99th=[16057]
   bw (  KiB/s): min=219768, max=467904, per=24.99%, avg=341061.83, stdev=110144.68, samples=477
   iops        : min=54942, max=116976, avg=85265.43, stdev=27536.16, samples=477
  lat (usec)   : 10=98.49%, 20=1.47%, 50=0.02%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.01%
  cpu          : usr=8.09%, sys=91.08%, ctx=11915, majf=0, minf=4
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=20479728,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=1333MiB/s (1398MB/s), 1333MiB/s-1333MiB/s (1398MB/s-1398MB/s), io=78.1GiB (83.9GB), run=60014-60014msec

Disk stats (read/write):
  nvme0n1: ios=20435311/0, merge=0/0, ticks=119271/0, in_queue=73143, util=83.08%
nvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1

QD=16:

Jobs: 16 (f=16): [r(16)][100.0%][r=2447MiB/s,w=0KiB/s][r=626k,w=0 IOPS][eta 00m:00s]
nvme: (groupid=0, jobs=16): err= 0: pid=2359: Sat Dec 12 00:06:27 2020
   read: IOPS=626k, BW=2446MiB/s (2565MB/s)(143GiB/60002msec)
    clat (usec): min=6, max=1219, avg=24.75, stdev=10.62
     lat (usec): min=6, max=1220, avg=24.88, stdev=10.62
    clat percentiles (nsec):
     |  1.00th=[ 7008],  5.00th=[ 8096], 10.00th=[15040], 20.00th=[17536],
     | 30.00th=[18304], 40.00th=[20096], 50.00th=[22144], 60.00th=[25216],
     | 70.00th=[28032], 80.00th=[32384], 90.00th=[38656], 95.00th=[46336],
     | 99.00th=[57088], 99.50th=[59648], 99.90th=[69120], 99.95th=[71168],
     | 99.99th=[82432]
   bw (  KiB/s): min=131350, max=188649, per=6.26%, avg=156867.63, stdev=15261.05, samples=1914
   iops        : min=32837, max=47162, avg=39216.68, stdev=3815.25, samples=1914
  lat (usec)   : 10=6.41%, 20=33.58%, 50=57.46%, 100=2.55%, 250=0.01%
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=2.88%, sys=26.37%, ctx=35034508, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=37576952,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=2446MiB/s (2565MB/s), 2446MiB/s-2446MiB/s (2565MB/s-2565MB/s), io=143GiB (154GB), run=60002-60002msec

Disk stats (read/write):
  nvme0n1: ios=37492849/0, merge=0/0, ticks=639710/0, in_queue=627138, util=100.00%

QD=64:

Jobs: 64 (f=64): [r(64)][100.0%][r=3620MiB/s,w=0KiB/s][r=927k,w=0 IOPS][eta 00m:00s]
nvme: (groupid=0, jobs=64): err= 0: pid=2266: Sat Dec 12 00:04:13 2020
   read: IOPS=926k, BW=3615MiB/s (3791MB/s)(212GiB/60009msec)
    clat (usec): min=6, max=2487, avg=68.38, stdev=23.99
     lat (usec): min=6, max=2487, avg=68.51, stdev=23.99
    clat percentiles (usec):
     |  1.00th=[    9],  5.00th=[   25], 10.00th=[   35], 20.00th=[   48],
     | 30.00th=[   58], 40.00th=[   66], 50.00th=[   73], 60.00th=[   78],
     | 70.00th=[   83], 80.00th=[   89], 90.00th=[   97], 95.00th=[  104],
     | 99.00th=[  117], 99.50th=[  121], 99.90th=[  129], 99.95th=[  133],
     | 99.99th=[  157]
   bw (  KiB/s): min=42690, max=67406, per=1.57%, avg=57985.88, stdev=4796.31, samples=7677
   iops        : min=10672, max=16851, avg=14496.18, stdev=1199.09, samples=7677
  lat (usec)   : 10=1.27%, 20=2.04%, 50=18.98%, 100=70.60%, 250=7.11%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=1.04%, sys=4.19%, ctx=54857445, majf=0, minf=64
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=55539634,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3615MiB/s (3791MB/s), 3615MiB/s-3615MiB/s (3791MB/s-3791MB/s), io=212GiB (227GB), run=60009-60009msec

Disk stats (read/write):
  nvme0n1: ios=55410046/0, merge=0/0, ticks=2830935/0, in_queue=3784646, util=100.00%

QD=256:

Jobs: 256 (f=256): [r(256)][100.0%][r=3975MiB/s,w=0KiB/s][r=1018k,w=0 IOPS][eta 00m:00s]
nvme: (groupid=0, jobs=256): err= 0: pid=2395: Sat Dec 12 00:09:44 2020
   read: IOPS=1019k, BW=3980MiB/s (4173MB/s)(233GiB/60002msec)
    clat (usec): min=4, max=4242, avg=250.52, stdev=96.45
     lat (usec): min=4, max=4242, avg=250.65, stdev=96.45
    clat percentiles (usec):
     |  1.00th=[   40],  5.00th=[   76], 10.00th=[  109], 20.00th=[  161],
     | 30.00th=[  206], 40.00th=[  239], 50.00th=[  262], 60.00th=[  285],
     | 70.00th=[  306], 80.00th=[  334], 90.00th=[  371], 95.00th=[  396],
     | 99.00th=[  437], 99.50th=[  449], 99.90th=[  474], 99.95th=[  486],
     | 99.99th=[  594]
   bw (  KiB/s): min=10416, max=23302, per=0.39%, avg=15943.97, stdev=2837.25, samples=30632
   iops        : min= 2604, max= 5825, avg=3985.73, stdev=709.32, samples=30632
  lat (usec)   : 10=0.44%, 20=0.12%, 50=1.19%, 100=6.87%, 250=36.19%
  lat (usec)   : 500=55.16%, 750=0.03%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=0.26%, sys=0.85%, ctx=60814904, majf=0, minf=255
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=61128161,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3980MiB/s (4173MB/s), 3980MiB/s-3980MiB/s (4173MB/s-4173MB/s), io=233GiB (250GB), run=60002-60002msec

Disk stats (read/write):
  nvme0n1: ios=60963109/0, merge=0/0, ticks=12313474/0, in_queue=16236847, util=100.00%