Adding more assembly to AARCH64 (macOS)

[lundman]

Looking into adding some possible speed^ups for macOS port on M1 etc, and rather than working in a vacuum I thought I'd run it by everyone. Someone might already have looked into it, or there are other plans in the workds.

I was tempted by the aes-gcm-armv8-unroll8_64.S but there isn't an easy way to plug in the whole aes-gcm that I see. So I started smaller, modeling after the aes_aesni.S work.

I picked aesv8_armx.S at random - honestly, it isn't clear to me which of the aarch64 options is the "newer"/"faster".

Confirmed valid with a hack call from zdb. Confirmed it runs in kernel. Not confirmed that it is faster though, I need more than 1 GB file pool to test that.

Working on this branch:
https://github.com/openzfsonosx/openzfs-fork/commits/unify_arm64

Specifically, this commit (while it exists)
openzfsonosx@4512a58

Thoughts?
@AttilaFueloep @mcmilk @ryao @behlendorf

[lundman]

Copying 50G of randomfile.bin

icp_aes_impl=generic 0.12s user 42.80s system 20% cpu 3:33.86 total
icp_aes_impl=aesv8   0.08s user 26.20s system 34% cpu 1:16.25 total

 icp_aes_impl=generic READ: bw=239MiB/s (250MB/s), 239MiB/s-239MiB/s 
          (250MB/s-250MB/s), io=69.9GiB (75.1GB), run=300003-300003msec

 icp_aes_impl=aesv8   READ: bw=566MiB/s (593MB/s), 566MiB/s-566MiB/s 
          (593MB/s-593MB/s), io=166GiB (178GB), run=300001-300001msec

So it is at least a little faster.

[ryao]

I am somewhat concerned that OpenSSL appears to be doing 5.4GB/sec on the M1 while this code doing 566MB/sec. It might be worthwhile looking into why there is an order of magnitude discrepancy, especially if we want to run as fast or faster on the M1 as we do on amd64 hardware.

[ryao]

We currently do not have assembly for accelerating GCM with AARCH64, so the door is open to any PR that contains working code that is more performant than the generic code and is reasonably clean such that it can pass review. Once such a PR has been merged, an alternative solution, such as grabbing aes-gcm-armv8-unroll8_64.S, would need to be evaluated on merit against what has already been merged.

That being said, I have a few thoughts:

1. For a generic solution, it would be nice if it were cross platform compatible between FreeBSD, Linux, MacOS and Windows.
2. If we are only concerned about accelerating AES on Apple silicon, perhaps we should look into whether there is any hardware encryption capability analogous to AES-NI. So far, I have only read about inline encryption that we would be unable to use inside ZFS, but geekbench reports a ridiculous 10.1GB/sec for single threaded AES-XTS on the M1, so maybe there is a capability that the CPU can use, although I would hope it applies to more than just XTS, which we currently do not support. Also, if we only initially care about improving performance on apple silicon, I am okay with that.
3. Implementations using SIMD and an AES-NI equivalent could co-exist with one another and could be done independently. Whichever you do, whether it is just one or both, is up to you. The idea to look into using an AES-NI equivalent on Apple silicon is just a suggestion of a possibility for making this even faster.
4. I see that you mentioned GCM and modified, module/icp/algs/modes/gcm.c, yet aes_benchit() is using ZIO_CRYPT_AES_256_CCM. That function also hard coded zkey.zk_crypt = 5; for decryption for no apparent reason. I am confused how your benchmark numbers show a difference, since ZIO_CRYPT_AES_256_CCM should not have been touched by what you did. Perhaps the version in your branch is not what you used to generate the benchmark numbers.

Lastly, I did a quick comparison on a Ryzen 7 5800X using OpenSSL 1.1.1s. The following disables PCLMULQDQ, AVX, AVX2 and AES-NI:

$ OPENSSL_ia32cap="~0x201200000200000000" openssl speed -evp aes-256-gcm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     150295.89k   171563.48k   177701.03k   179993.26k   180259.50k   180458.84k

The following disables only AVX, AVX2 and AES-NI:

$ OPENSSL_ia32cap="~0x201200000000000000" openssl speed -evp aes-256-gcm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     216570.10k   242423.29k   245628.50k   246599.68k   246494.55k   246344.36k

The following disables only AES-NI:

$ OPENSSL_ia32cap="~0x200000000000000" openssl speed -evp aes-256-gcm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     214910.79k   241169.45k   246998.53k   247417.86k   247728.81k   247851.69k

The following uses AES-NI:

$ openssl speed -evp aes-256-gcm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm     601403.16k  1595267.17k  3160675.07k  4066395.48k  4621901.82k  4625645.57k

Also, just in case you really did benchmark CCM rather than GCM, here are some CCM numbers. The following disables PCLMULQDQ, AVX, AVX2 and AES-NI:

$ OPENSSL_ia32cap="~0x201200000200000000" openssl speed -evp aes-256-ccm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-ccm      61544.40k   103538.56k   125276.16k   132344.83k   134575.45k   134769.32k

The following disables only AVX, AVX2 and AES-NI:

$ OPENSSL_ia32cap="~0x201200000000000000" openssl speed -evp aes-256-ccm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-ccm      61055.56k   103736.21k   125682.60k   132888.49k   134564.52k   134774.78k

The following disables only AES-NI:

$ OPENSSL_ia32cap="~0x200000000000000" openssl speed -evp aes-256-ccm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-ccm      61335.05k   103785.90k   125605.46k   132507.99k   134613.67k   134671.02k

The following uses AES-NI:

$ openssl speed -evp aes-256-ccm
...
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-ccm     205449.56k   538069.66k   835824.21k   971865.09k  1020491.09k  1024251.22k

Oddly, it looks like OpenSSL 1.1.1s does not have an AVX/AVX2 implementation for aes-256-gcm or aes-256-ccm. It also appears to have no implementation to take advantage of PCLMULQDQ with SSE2 on aes-256-ccm. I am not sure whether SSE2 or SSE2 + PCLMULQDQ is equivalent to ARM's 128-bit SIMD to make the comparison fair, but these numbers suggest that does not really matter, given that the Apple numbers for a generic implementation outperform the OpenSSL numbers for everything but AES-NI on Zen 3. The assembly routine just makes it even better, although it is still significantly slower than AES-NI, which is why it might be worthwhile looking into using Apple's hardware encryption instructions if they exist.

Although it should be well known at this point, these numbers confirm that AES GCM is faster than AES CCM. If you really did benchmark ZIO_CRYPT_AES_256_CCM, you probably should redo your benchmark with ZIO_CRYPT_AES_256_GCM and make certain that the code really is writing the one that is being specified. Presumably, if you did benchmark CCM instead of GCM, then revised benchmarks should show the performance on Apple silicon to be even more amazing.

[ryao]

I was told in IRC that on the M1, Linux reports these CPU features: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint. That has aes, so presumably, there is some kind of AES acceleration support. I would hope that aes there means that aes is supported under the ARM cryptographic extensions, but I do not have time to check.

[AttilaFueloep]

Some nit-picking comments on the above.

Oddly, it looks like OpenSSL 1.1.1s does not have an AVX/AVX2 implementation for aes-256-gcm

It does, actually that's what we use for AVX support, see 31b160f (#9749).

It also appears to have no implementation to take advantage of PCLMULQDQ with SSE2 on aes-256-ccm

AES-CCM uses a CBC-MAC so there's no need to use CLMUL.

[ryao]

Some nit-picking comments on the above.

Oddly, it looks like OpenSSL 1.1.1s does not have an AVX/AVX2 implementation for aes-256-gcm

It does, actually that's what we use for AVX support, see 31b160f (#9749).

It does not appear to be being used based on the test results. Either that or the feature bitmask failed to disable AVX/AVX2 support.

[AttilaFueloep]

The openssl AVX AES-GCM implementation requires AES-NI and PCLMULQDQ support since GHASH, encryption and counter processing are interwoven for performance reasons. And in real live there is no AVX CPU without AES-NI and PCLMULQDQ so it wouldn't be reasonable not to use them.

[mcmilk]

Apple Mac Mini M1 (16K PAGESIZE kernel)

$ openssl speed -evp aes-256-gcm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-256-GCM     802303.24k  2224526.34k  3618944.43k  4987108.69k  5426866.86k  5466024.62k

$ openssl speed -evp aes-256-ccm
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-256-CCM     319222.75k   499233.79k   734914.39k   777885.70k   791183.36k   789118.98k

$ cat /proc/cpuinfo
fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma \
lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint

[ryao]

OpenSSL is definitely using hardware acceleration to get that level of performance.

[mcmilk]

Yes, I know these ... and I cold add support for that - but the interfaces we have currently, are not ready for such things :(

When I should do some more work on it, I would make use of the include/sys/zfs_impl.h interface and so on - like for SHA2 and BLAKE3.
Also PPC64 has SIMD for AES ... even these we could support. In some days I am getting some virtual machines for aarch64 + ppc64... directly for OpenZFS - I hope we have then action runners on that arch :)

One qustion I have is, how can we make use of crypto routines provided by the OS - FreeBSD, Windows and MacOS have such things.

[ryao]

how can we make use of crypto routines provided by the OS - FreeBSD, Windows and MacOS have such things.

With great pain. They have different kernel and userspace APIs, unlike OpenSolaris, so ztest would no longer be able to catch bugs in the cryptographic functions that could occur in kernel space. Subtle bugs could exist in one platform’s code that would go uncaught until it has created a headache for us.

[mcmilk]

okay, then we should go the way we already started to go ... with some algos at icp which does the things ...

[AttilaFueloep]

but the interfaces we have currently, are not ready for such things

Yes, but the main problem is that we have two different kinds of gcm asm implementations, one uses the existing C implementation (pclmulqdq), uses asm only for GHASH and relies on icp_aes_impl to do the encryption. The other does GHASH, encryption and counter processing in a single asm routine (avx, sse4_1). The first one is quite slow, while the second one is about an order of magnitude faster.

I'm planning to refactor the selection of implementations but due to the intervening of the AES and GCm code, I decided to not do it in #14531 as it would make reviewing much more involved.

[lundman]

ccm.c is somewhat simple file, just calls the encrypt function passed to it, which leads you back to aes_impl.c - for some reason the selector to the aes implementation to use lives in gcm.c. Like I said, I just copied whatever aesni.S does, renaming to aesv8.S.

So I used ZIO_CRYPT_AES_256_CCM as it will use and test the underlying aes currently selected, to get a better reading of performance. GCM is the same thing, just more complicated iv computation with ghash and all that. gcm should also have been improved by this change, but perhaps not as visibly due to the additional overhead.

As you can see in aesv8_armx.S it is using the new fancy aese v6.16b,v0.16b mnemonics. For non mac platforms, it might need extra code for v7, neon, and v8 capability checks. . Non-mac arm64 will most likely want aesv7_armx.S as well. All mac's are v8, so I dont need to check for v7 or neon.

But as mentioned, I think this could do with the zfs_impl that has been started, so that one can make current gcm/ccm as the "generic" implementation, and additionally, plug in fully featured versions like aes-gcm-armv8-unroll8_64.S . I will wait for that, adding aes is about all I can do in this area (maaaybe add ghash ... as that seems to have separate code)/file.

Meanwhile, I am adding armv8_sha256.S and armv8_sha512.S as those should slot in easily.

I personally am not that interested in using macOS/Windows own/built in crypto work, as it is often restrictive. Both as in linking, and might-not-support-all-the-things-we-want. (say blake3 for example). I also like using zdb to test the code in userland first. None of my platforms have the same API for kernel and userland. So are you truly testing the same code then? At least I don't have to worry about GNU licenses :)

But with a zfs_impl setup, that certainly could still be done, an additional option to use OS native. But I'd prefer ZFS have fast versions built-in.

In a prior commit I test for the CPU capabilities of AESV8 here:
openzfsonosx@f1ff8e9#diff-fc8876814919c610c055726a5db3fc2a2756b6a82824793ae9d80f42f1f1d494R100-R104

so you can think of it as the aesni equivalent on arm64. armv7, and/or, neon, are baked into v8 - so no need to test for them

[ryao]

I mostly just mention armv7 with the assumption it would be something Linux wants, I have no need of it, so I'm happy to not talk about armv7 :)

I usually am the one to care most about portability and I do not see any reason to care here right now. No users are asking for it. ZFS likely has very limited usage on armv7 Linux machines and that is unlikely to ever change, so I highly doubt any users will ever ask for it.

That said, implementing hardware AES acceleration on ARMv8 might kill much of the case for eventually merging #14249. That was being done on the assumption that lower specification hardware like the raspberry pi 4 will never get fast AES encryption, but this could change that. @robn What are your thoughts on the usefulness of chacha20-poly1305 encryption if the Raspberry Pi were to get fast AES?

[AttilaFueloep]

Yeah, as I wrote above, the performance gains of just using hardware accelerated AES and CLMUL are quite moderate. The real performance gain comes from keeping the pipelines filled and utilizing the different instruction units in parallel. To do that you have to do all processing (GHASH, encryption, XOR, CTR) in a single routine and you have to operate on larger chunks of data. Judging from the name, aes-gcm-armv8-unroll8_64.S may do exactly that.

I'm planning to refactor the implementation selection once #14531 is done. This should allow for easier plugin in of such implementations. I'll open a discussion where we can talk about the details once I'm ready.

[lundman]

Awesome, I'm happy to wait for that. Adding aesv8 for macOS will be enough for the current users I think, even if it doesn't go upstream.

[danieldjewell]

For what it's worth -- IMHO, I personally don't think that the availability of hardware AES crypto should be a factor on whether or not to include Chacha20-Poly1305 support (#14249) ... Just because there is fast AES, doesn't necessarily obviate the benefits of having the option to use Chacha20. (FWIW and AFAIK, Chacha20 has pretty good performance regardless of hardware accelerations - there could still be cases where that is faster/better for the end-user regardless of the availability of hardware AES crypto acceleration.)

[mcmilk]

I think OpenZFS will have both (ChaCha20 and AES) in the near future.... and both will also use some assembly.

[lundman]

This is a "can lundman really put in openssl assembly and make it work" proof-of-concept, and I am emboldened by the success I have got.

I was unaware of this: #12171

So it would be straight forward to add in armv7 as well, then do some CPUID checks on capabilities to work out which one to call.

Have we decided on how the ARM CPUID will be done (in ZFS)?

Are we just exposing ID-AA64ISAR0-EL1 (etc) internally, and testing against those bits, (AES, bit 4-7)
or
are we converting these to internal bit-map, for ZFS-AES-enabled, ZFS-SHA256-enabled etc ?

How does it work with ArmV7? How does Linux prefer to do that?

I additionally tested sha256-armv8 and sha512-armv8.

sha256-generic:
READ: bw=468MiB/s (491MB/s), 468MiB/s-468MiB/s (491MB/s-491MB/s), io=137GiB (14\
7GB), run=300001-300001msec

sha256-armv8:
READ: bw=904MiB/s (948MB/s), 904MiB/s-904MiB/s (948MB/s-948MB/s), io=265GiB (28\
4GB), run=300001-300001msec

sha512-generic:
READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=156GiB (16\
8GB), run=300001-300001msec

sha512-armv8:
READ: bw=954MiB/s (1000MB/s), 954MiB/s-954MiB/s (1000MB/s-1000MB/s), io=280GiB \
(300GB), run=300001-300001msec

[lundman]

Why are you guys using my benchmark in a real situation? I created a file-based pool, ran a short fio to make sure it doesn't catch on fire.

Actually, how do you guys get that nice "all impls" output.

[lundman]

@lundman - armv8 for sha256 and sha512 are implemented in openzfs - only some simd.h magic might be open @rincebrain did also testings, see #13741:

I did not know that, certainly not in my tree - but clearly in openzfs. I must be missing a commit somewhere. I'll pull it in and check how you guys decided to do the CPUID work and emulate it.

[mcmilk]

@lundman - armv8 for sha256 and sha512 are implemented in openzfs - only some simd.h magic might be open @rincebrain did also testings, see #13741:

I did not know that, certainly not in my tree - but clearly in openzfs. I must be missing a commit somewhere. I'll pull it in and check how you guys decided to do the CPUID work and emulate it.

It's called micro benchmark and it's done via module/zfs/zfs_chksum.c
The same thing is used here for years: module/zcommon/zfs_fletcher.c

[ryao]

Why are you guys using my benchmark in a real situation? I created a file-based pool, ran a short fio to make sure it doesn't catch on fire.

That explains the low numbers.

Actually, how do you guys get that nice "all impls" output.

It is from running a Linux VM. A file in /proc/spl/kstat/zfs contains benchmark data for the existing implementations.

[lundman]

Oh it's a PR, ok, sure, that's why it isn't in there. Lemme fetch PR, and build on it.

ryao: M1 macbook ventura, and M2 macmini ventura

[lundman]

OK I ported over the PR work, and massaged it to work for macOS

# sysctl kstat.zfs.misc.chksum_bench.verbose=1
kstat.zfs.misc.chksum_bench.verbose: 0 -> 1

#  sysctl kstat.zfs.misc.chksum_bench         
kstat.zfs.misc.chksum_bench.chksum_bench: 
implementation               1k      4k     16k     64k    256k      1m      4m     16m
edonr-generic               504     566     584     591     586     586     585     580
skein-generic               120     127     129     129     129     127     129     129
sha256-generic              112     119     118     117     117     118     119     121
sha256-armv7                336     359     366     367     368     368     366     366
sha256-neon                 345     369     376     377     378     378     377     377
sha256-armv8-ce            1884    2215    2315    2343    2337    2352    2310    2320
sha512-generic              169     186     192     194     193     194     194     193
sha512-armv7                503     562     580     581     585     585     583     578
sha512-armv8-ce            1120    1303    1359    1374    1377    1378    1376    1352
blake3-generic               72      69      68      68      68      68      68      67
blake3-sse2                 403    1341    1446    1395    1486    1478    1441    1375
blake3-sse41                428    1390    1511    1584    1549    1540    1480    1461

I'll post to the PR the changes I've needed to do

[ryao]

It is an odd just how bad the generic numbers are for your machine in comparison to @mcmilk’s machine. That said, the optimized assembly numbers match, which is good.

[lundman]

Nah, to be able to lldb with the kernel, I use 'CFLAGS=-g -O0'. So not too worried about that, yet.

[lundman]

For completeness sake, -O2

kstat.zfs.misc.chksum_bench.chksum_bench: 
implementation               1k      4k     16k     64k    256k      1m      4m     16m
edonr-generic              2665    3095    3261    3317    3335    3340    3328    3321
skein-generic               269     284     288     288     287     288     287     284
sha256-generic              270     286     291     292     291     291     291     291
sha256-armv7                343     361     366     367     367     367     366     367
sha256-neon                 353     371     376     377     377     377     377     377
sha256-armv8-ce            2101    2287    2332    2351    2354    2354    2310    2319
sha512-generic              405     449     462     465     464     458     462     458
sha512-armv7                495     566     583     583     586     575     587     576
sha512-armv8-ce            1211    1288    1357    1291    1319    1339    1385    1369
blake3-generic              580     566     569     538     550     558     559     555
blake3-sse2                 445    1392    1519    1579    1532    1472    1466    1487
blake3-sse41                450    1431    1585    1660    1642    1646    1637    1615

[danieldjewell]

It still never ceases to amaze me how much of an improvement the crypto extensions (of any architecture) provides... on SHA256, roughly 700-800% improvement in raw bandwidth? (Which, theoretically, would also mean lower overall CPU usage - and battery usage...)

Kudos!