Ask for help on disk write data, zfs write data and performance issues through nfs

[lenghenglong]

After installing zfs on the storage device, the performance of pure writing to lun and the performance of pure writing to the zfs file system in the case of raid6 and raid0 were tested respectively
Here is my test environment:
Storage device: OS memory 64G, physical memory 768G
zfs version: zfs-215
vdbench version: vdbench50407
Part of the script written 100% to lun(16 dir):
wd=wd1,sd=sd*,xfersize=8k,rdpct=0,seekpct=random,openflags=o_direct
rd=run1,wd=wd1,iorate=max,elapsed=100,warmup=30,interval=1,threads=6

Part of the script to write data to zfs100%:
fsd=default,depth=3,width=8,files=1000,size=8k,openflags=o_direct
fwd=fwd1,fsd=fsd*,operation=write,rdpct=100,xfersize=8k,fileio=random,fileselect=random,threads=48

zpool: 16 4TB data luns + 1 20G special lun +1 10G log lun (recordsize=128K)

The test results are as follows:
raid6：lun iops rate= 97000 resp=0.978ms ; zfs iops rate=41700 resp=1.101ms
raid0：lun iops rate=530000 resp=0.174ms ; zfs iops rate=42000 resp=1.072ms
During this process, I observed that the delay of the special lun is around 2~6ms, and sometimes it reaches 13ms, while other data luns are below 1ms.
After that, the file system was mounted through nfs to perform write operations, and the iops was reduced to about 20,000. Comparing the zfs file system locally with 16 ext4s created on 16 4TB luns, the iops of zfs is 45,000, while the iops of ext4 is 170,000, the gap is very large.
I have a few questions about this and hope someone can guide me:

1. Why is the lun performance of raid0 improved so much compared to raid6, while the performance of zfs has no obvious change?
2. Why is the latency of special lun so much higher than that of data lun?
3. Why does the performance of zfs drop so much after the data passes through nfs? Is it related to the way nfs is written? For example, a large number of getattr operations, will there be a bottleneck on zfs?

[rincebrain]

I would recommend you specify things about the storage you used for each "disk" you gave ZFS, either physical disk or whatever abstraction on top of it, if you would like useful answers.

[lenghenglong]

Sorry that the pool was destroyed at that time, I only found this information, this is the screenshot printed by zpool iostat during the vdbench test, but other parameters of the pool are default.
16 data luns: alloc 3.95G free 3.98T
speacial lun: alloc 2.58G free 16.9G
logs lun: alloc 134M free 9.37G

[amotin]

NFS writes usually enforce some cache flush semantics: either every request is marked as synchronous, or cache flush requests are inserted regularly. ZFS strictly implements the requested write cache semantics, that is why NFS writes can be slower than for some other file systems, and/or may need SLOG device for better performance.

[lenghenglong]

Thank you for your reply, I will learn more about it and test it later.