Docker provider performance

[deviantintegral]

Thanks for the writeup at https://ddev.com/blog/docker-performance-2023/ - it's great to see the improvements everywhere in this area.

I was surprised to see such a gap between colima and everything else and wanted to see more. I ran some tests on my M2 Max.

On reproducibility:

• I have 2TB of storage, and larger drives have faster IO performance, so sharing what you have would be useful.
• I didn't see any info on settings for OrbStack, but out of the box it allows "max" CPU usage. It shows that for me as 1200%, so I'm guessing that is 12 cores. While I doubt CPU contention is an issue, should that be reduced to 600% for comparisons with colima being set to 6?
• I tested with the script at https://github.com/ddev/ddev-puppeteer. Nice! It'd be great to see something set up to work with hyperfine too, since it can rerun until it gets a stable result. I wonder if using drush site install would be simpler for users to test with? Then they could run hyperfine --export-csv out.csv ./bench.sh to easily include in spreadsheets.

I didn't isolate my system from running other processes, but I did have backups and the like paused.

• Colima using mutagen: 16.5s.
• Colima using virtiofs: 26.5s this is significantly different than your results
• OrbStack + mutagen: 14.5s
• OrbStack virtiofs: 18.5s

I wonder if something is up with your colima benchmark results when not using mutagen. There shouldn't be such a stark difference between our systems.

A 30% improvement over stock virtiofs is pretty good for OrbStack. It will be worth watching to see if their custom dynamic caching and optimizations (from https://docs.orbstack.dev/architecture#file-sharing) are subject to any consistency bugs in the long term.

I was having other troubles... until I realized I was running colima with sshfs instead of virtiofs. I wonder, could ddev detect when sshfs is used and the performance node is none, and warn about it? It may save some support requests in the future.

[rfay]

Thanks! I also have 2TB, but MBA 2020.

It does seem that OrbStack has its own adjustments for resources.

SSHFS has traditionally been much faster than other mount types, but now that VirtioFS is maturing, it will probably be OK to suggest it in the future. But as you know... it just can't work without mutagen.

Thanks for looking at this! The assumption is that a d10 site is set up and running before using the ddev-puppeteer. Improvements to that are very welcome.

[deviantintegral]

I just pushed up https://github.com/deviantintegral/ddev-d10perf. It should just be a git clone and then ./test.sh, assuming you have both colima and orbstack installed. I don't have docker desktop installed at the moment, and tbh I'm not sure if it can be controlled at the CLI like this.

Output from my machine:

Benchmark 1: echo colima-mutagen; ddev drush site:install demo_umami -y
  Time (mean ± σ):      9.532 s ±  0.062 s    [User: 0.069 s, System: 0.067 s]
  Range (min … max):    9.398 s …  9.626 s    10 runs

Benchmark 2: echo colima-virtiofs; ddev drush site:install demo_umami -y
  Time (mean ± σ):     14.908 s ±  0.354 s    [User: 0.070 s, System: 0.069 s]
  Range (min … max):   14.422 s … 15.560 s    10 runs

Benchmark 3: echo orb-mutagen; ddev drush site:install demo_umami -y
  Time (mean ± σ):     10.299 s ±  0.140 s    [User: 0.069 s, System: 0.072 s]
  Range (min … max):   10.165 s … 10.525 s    10 runs

Benchmark 4: echo orb-virtiofs; ddev drush site:install demo_umami -y
  Time (mean ± σ):     13.811 s ±  0.561 s    [User: 0.071 s, System: 0.071 s]
  Range (min … max):   13.393 s … 15.237 s    10 runs

Summary
  echo colima-mutagen; ddev drush site:install demo_umami -y ran
    1.08 ± 0.02 times faster than echo orb-mutagen; ddev drush site:install demo_umami -y
    1.45 ± 0.06 times faster than echo orb-virtiofs; ddev drush site:install demo_umami -y
    1.56 ± 0.04 times faster than echo colima-virtiofs; ddev drush site:install demo_umami -y

The csv:

command,mean,stddev,median,user,system,min,max
echo colima-mutagen; ddev drush site:install demo_umami -y,9.532388174200001,0.06173071975500223,9.542991853699998,0.06907176,0.06747492000000001,9.3979361037,9.6257756037
echo colima-virtiofs; ddev drush site:install demo_umami -y,14.908437269999999,0.353703431341317,14.8798045617,0.06988565999999999,0.06886882000000001,14.4224231447,15.5600041867
echo orb-mutagen; ddev drush site:install demo_umami -y,10.299467811799998,0.13968897686197523,10.2320033327,0.06938416,0.07177592000000002,10.1653064777,10.5246463537
echo orb-virtiofs; ddev drush site:install demo_umami -y,13.8106825117,0.5608570990404511,13.6107940197,0.07086876,0.07100992,13.3933592707,15.2369909777

[rfay]

So it's basically doing a drush si? I measured drush si in previous rounds of this but didn't think they were as meaningful as actual web install. Nice way to get comparative numbers though. The key experience of developers (mostly) though is how fast php-fpm can perform against PHP files, and although drush si does some that way I don't think it does as much as a web install.

[deviantintegral]

Thinking through this:

• A web install has the advantage of acting like a developer in most cases - running code and clicking through a web browser on their host.
• It has the advantage of PHP's opcache, PHP itself should be faster.
• However, drush doesn't have to re-bootstrap Drupal or the installer so many times.
• The web installer has to poll the batch api, so there could be more variance in timings.

This all has me thinking that the best test case would be running Drupal core FunctionalJavascript tests - it runs the installer at the CLI (well, in PHPUnit) but then hits the site with a browser.

I also realized with the above repo that it was pretty easy for me to take the scripts and have them run Playwright tests I already have going. It's not 1-1 (we use sqlite, we only install the site once and then copy for each test case), but it does allow for relative timings. It also is a much more complex site with many more files on disk, so we can see if that affects things.

Benchmark 1: echo colima-mutagen; ./run.sh
  Time (mean ± σ):     62.103 s ±  1.066 s    [User: 0.191 s, System: 0.199 s]
  Range (min … max):   60.679 s … 63.396 s    10 runs

Benchmark 2: echo colima-virtiofs; ./run.sh
  Time (mean ± σ):     138.323 s ±  2.624 s    [User: 0.188 s, System: 0.203 s]
  Range (min … max):   133.982 s … 143.042 s    10 runs

Benchmark 3: echo orb-mutagen; ./run.sh
  Time (mean ± σ):     61.841 s ±  0.547 s    [User: 0.203 s, System: 0.218 s]
  Range (min … max):   61.033 s … 62.719 s    10 runs

Benchmark 4: echo orb-virtiofs; ./run.sh
  Time (mean ± σ):     98.994 s ±  2.204 s    [User: 0.190 s, System: 0.217 s]
  Range (min … max):   96.614 s … 104.267 s    10 runs

Summary
  echo orb-mutagen; ./run.sh ran
    1.00 ± 0.02 times faster than echo colima-mutagen; ./run.sh
    1.60 ± 0.04 times faster than echo orb-virtiofs; ./run.sh
    2.24 ± 0.05 times faster than echo colima-virtiofs; ./run.sh

command,mean,stddev,median,user,system,min,max
echo colima-mutagen; ./run.sh,62.10256130169999,1.0663075803007815,62.002203755699995,0.19130574,0.19942796,60.6786364222,63.396390923199995
echo colima-virtiofs; ./run.sh,138.3229798516,2.624381637498447,137.50304336020002,0.18810084,0.20286186,133.98208979720002,143.04158900520002
echo orb-mutagen; ./run.sh,61.84111375599999,0.5465674233141566,61.7271177982,0.20288804,0.21792976000000003,61.0326263392,62.7189167142
echo orb-virtiofs; ./run.sh,98.99394212249999,2.203847663244651,98.4449614017,0.18987384000000002,0.21660246,96.6136127142,104.2669074642

My takeaways:

• With mutagen, colima and orbstack are basically identical.
• With virtiofs, orb is clearly better than colima.
• But given the above represents a real site (and not just Drupal core), I think it's good we still default to mutagen.

[rfay]

Great work. I agree that Drupal functional tests would be a great way to go.

I also agree about Mutagen for now, but it may be that OrbStack and DD will eventually make it so we don't need mutagen for local dev.

[kinglozzer]

Just dropping in here to note this update from OrbStack: https://orbstack.dev/blog/fast-filesystem. I haven’t conducted any proper benchmarking like you have above, but apachebench runs against my PHP app’s homepage with performance_mode: 'mutagen' and performance_mode: 'none' yield near-identical results now.

[rfay]

Orbstack was essentially already equivalent in https://ddev.com/blog/docker-performance-2023/, it's just so good.