Build Performance Results

This is a comparison between:

• Visual Studio 15.8.4, Xamarin.Android 9.0.x
• Visual Studio 15.9 Preview 3, Xamarin.Android 9.1.0.x - soon to be released as Visual Studio 2017 15.9.0 stable
• Visual Studio 16.0 Preview 2, Xamarin.Android 9.1.199.x - soon to be released as Visual Studio 2019* or the next version of Visual Studio "Dev16".

Our core focus here is improving the developer loop, in cases such as:

• You have a currently built/deployed Xamarin.Forms app.
• You change one line of XAML in your NetStandard Xamarin.Forms project.
• You build/deploy again.

Along the way if we happened to find something to improve initial builds, that would be a bonus.

Projects Used

• Hello Forms: File | New Xamarin.Forms project with a NetStandard library. Using project template from 15.8.4.
• SmartHotel360: A sample Xamarin.Forms app with 99 NuGet packages, using a fork so we have updated Xamarin.Forms to 3.0.x.
• Xamarin.Forms Control Gallery: The "control gallery" sample project has around 7 Xamarin.Android library projects and includes building Xamarin.Forms itself.

Specific details about changes mades made in each version of Visual Studio can be found here.

MSBuild Targets Timed

• Build or build
• SignAndroidPackage or package, this creates the Android APK file and signs it
• Install or deploy, this deploys the APK to the attached device

I performed all of these timings with a USB-connected Pixel 2 device.

From the logs, what are first/second/third builds?

• First: simulates a build after an initial checkout, after nuget restore.
• Second: a build with no changes.
• Third: modify a XAML file, build.

Improvements to Incremental Build Times

App	15.8	15.8 Log	15.9	15.9 Log	16.0	16.0 Log
Hello Forms (no changes)	00:02.99	binlog	00:02.75	binlog	00:02.39	binlog
Hello Forms (XAML change)	00:08.03	binlog	00:06.63	binlog	00:04.34	binlog
SmartHotel360 (no changes)	00:03.74	binlog	00:03.54	binlog	00:02.52	binlog
SmartHotel360 (XAML change)	00:10.62	binlog	00:08.34	binlog	00:07.17	binlog
Xamarin.Forms (no changes)	00:16.48	binlog	00:11.20	binlog	00:12.40	binlog
Xamarin.Forms (XAML change)	00:52.53	binlog	00:35.34	binlog	00:36.16	binlog

NOTE: 16.0p2 has a performance regression in the ResolveSdks MSBuild task, as seen in the Xamarin.Forms project.

# 15.9
 80 ms  ResolveSdks                                7 calls
# 16.0
127 ms  ResolveSdks                                7 calls

This should be resolved by PR #2420 in the next 16.0 preview.

Improvements to Deploy Times

App	15.8	15.8 Log	15.9	15.9 Log	16.0	16.0 Log
Hello Forms (first)	00:15.22	binlog	00:12.90	binlog	00:06.95	binlog
Hello Forms (no changes)	00:03.07	binlog	00:02.83	binlog	00:02.14	binlog
Hello Forms (XAML change)	00:07.06	binlog	00:06.63	binlog	00:02.94	binlog
SmartHotel360 (first)	00:17.39	binlog	00:15.65	binlog	00:12.78	binlog
SmartHotel360 (no changes)	00:03.88	binlog	00:03.69	binlog	00:02.61	binlog
SmartHotel360 (XAML change)	00:09.03	binlog	00:05.10	binlog	00:04.03	binlog
Xamarin.Forms (first)	00:19.30	binlog	00:13.76	binlog	00:14.43	binlog
Xamarin.Forms (no changes)	00:16.82	binlog	00:11.90	binlog	00:10.91	binlog
Xamarin.Forms (XAML change)	00:21.21	binlog	00:15.07	binlog	00:14.33	binlog

Initial Build Times

We regressed very slightly on the initial build time in 15.9, but improved in 16.0.

Reviewing the build logs from 15.9, I believe this was due to #deletebinobj related fixes. These are fixes for "correctness" of our build, where developers could get a strange error that is resolved by deleting their bin and obj directories. I believe these fixes have added time to the build, but it is important to be correct before optimizing things. We also improved the first deployment times, so there is certainly some tradeoff here.

App	15.8	15.8 Log	15.9	15.9 Log	16.0	16.0 Log
Hello Forms (build)	00:35.93	binlog	00:37.33	binlog	00:27.81	binlog
Hello Forms (package)	00:07.74	binlog	00:07.33	binlog	00:06.80	binlog
Hello Forms (deploy)	00:15.22	binlog	00:12.90	binlog	00:06.95	binlog
SmartHotel360 (build)	01:04.20	binlog	01:05.90	binlog	00:53.30	binlog
SmartHotel360 (package)	00:10.65	binlog	00:10.53	binlog	00:10.16	binlog
SmartHotel360 (deploy)	00:17.39	binlog	00:15.65	binlog	00:12.78	binlog
Xamarin.Forms (build)	02:51.11	binlog	02:59.56	binlog	02:04.28	binlog
Xamarin.Forms (package)	00:18.70	binlog	00:18.19	binlog	00:16.49	binlog
Xamarin.Forms (deploy)	00:19.30	binlog	00:13.76	binlog	00:14.43	binlog

D8 / R8 Performance Comparison

Xamarin.Android has support starting in 16.0 Preview 2 for Google's new d8 and r8 dex compiler. In addition to supporting the feature set of d8, in some cases the _CompileToDalvik MSBuild target will have a 15-20% speed increase. This will add up to better initial build times.

The following times are using 16.0 Preview 2:

App	Before	Before Log	d8 / r8	d8 / r8 Log
Hello Forms (debug / default)	00:27.81	binlog	00:23.49	binlog
Hello Forms (release / proguard)	00:33.38	binlog	00:32.61	binlog
SmartHotel360 (debug / default)	00:53.30	binlog	00:48.76	binlog
SmartHotel360 (release / proguard)	01:06.06	binlog	01:03.79	binlog
Xamarin.Forms (debug / default)	02:04.28	binlog	01:57.71	binlog
Xamarin.Forms (release / proguard)	02:24.06	binlog	02:40.94	binlog

Aapt2 Performance Comparison

Xamarin.Android has had initial support for aapt2 starting in 15.9. The prospect is that using aapt2 will make incremental builds involving AndroidResource files much faster. This is a bigger benefit to "classic" Xamarin.Android apps, since they heavily rely on AndroidResource files. Xamarin.Forms apps will still see a benefit when AndroidResource files are modified in the Xamarin.Android head project.

In order to test this, I modified my scripts slightly to do the following:

• Time SignAndroidPackage, so Build and the APK creation happen at the same time.
• Incremental builds modify an AndroidResource file, instead of C# code/XAML.

Results from 16.0 Preview 2:

App	Aapt	Aapt Log	Aapt2	Aapt2 Log
Xamarin.Forms (initial)	02:16.04	binlog	02:04.67	binlog
Xamarin.Forms (no changes)	00:10.76	binlog	00:10.65	binlog
Xamarin.Forms (XML change)	00:40.98	binlog	00:38.42	binlog

NOTE: Further exploration is needed here, since all the projects tested in this document are Xamarin.Forms projects.

Other Notes

A build with no changes is generally 2-3 seconds, which is certainly improved from what it was 6 months ago. We are still working on this, as there are still a few MSBuild targets running on every build that could be skipped (or improved).

Since these timings run build, package, and deploy in three distinct steps, this overhead is present in all three builds.

There are some additional changes in master that will improve initial build times. These are still in progress, but should be coming soon! I'll update the numbers as they are available.

Lessons learned

In PR #2105, I was seeing a massive improvement to first builds of "Hello World" apps.

I realized since I was timing builds with only the OSS xamarin-android, I was basically timing builds without Fast Deployment. This feature sideloads assemblies (and is enabled by proprietary Xamarin.Android code), and so .NET assemblies are not embedded in the APK by default.

This meant that the PCL facade assemblies were being embedded in the APK during my timing. That explains the huge difference in build times I measured, since making Java.Interop a non-PCL enabled OSS xamarin-android to not embed as many assemblies.

However, this changes was still great, since it improved initial deployment times, as seen in the times above. We have also found developers common disable Fast Deployment for various reasons (they hit a bug?), so build times will be improved there.

Conclusion

Some notes on what we should always do when timing builds.

• Always start with a clean working copy! git clean -dxf
• Time Release builds of Xamarin.Android.Build.Tasks.dll.
• Time changes within the context of the proprietary Xamarin.Android code.
• If possible, potentially even deploy the changes to the system Xamarin.Android install.
• Time more than one project. Record everyting. Put it in the commit message!

It is possible we should look into some kind of CI setup / scripting to make these easier.