Problems in napp-it for Windows

[guenther-alka]

When I created the option to modify properties in my napp-it web-gui for Windows with:
zfs set atime=off tank/data or
zfs set readonly=on tank/data

the property is not set but i get
"property may be set but unable to remount filesystem"

Is this a known issue?

[lundman]

So on Solaris/illumos, you can modify the mount options (readonly etc) dynamically, but on Linux the mount options are only acted upon at mount. Linux added the ability to automatically unmount, then re-mount the dataset when (certain) properties are changed.
Windows are somewhat in between, and I have not got to the item in my task-list that deals with checking what options needs remounting (if any). So at the moment, the code that tries to remount is just commented out, and this is why you see that message.

You'll independently want to check that the property works, for example readonly - did I finish that code? The focus has been stability over "nice things" like readonly :) But for example, I know if you set driveletter, it definitely would need to remount, but does not (yet) and a manual unmount/mount trigger is needed (for now).

[sskras]

@lundman, thanks for very detailed explanation! That text is equivalent of a nice blog entry. :)

[guenther-alka]

Thanks.
I can live with that. Bsically I expected to "just work" like on Solaris.
At the moment I only display atime and readonly settings. You have to change manually.

What would be the correct sequence?
modify + unmount + mount or
unmount + modify + mount

For the driveletter I expected that behaviour and
set driveletter + unmount + mount recursively

And yes, care about stability over features.
If you know about, you can handle that with scripts or via cli

[lundman]

modify + unmount + mount or
unmount + modify + mount

no difference - but i would refer the first one, since you can use the property check, and only do the remount if it changed ok.

[guenther-alka]

Just to add
It is not enough to unmount a sub filesystem where you want to modify settings. You must unmount the parent pool filesystem, modify and remount recursively all filesystems

[lundman]

Oh yeah, it's real complicated with driveletter, all children have to also be remounted ;(

[guenther-alka]

The driveletter thing is really tricky

An average Windows user expects a ZFS pool ex tank to come up like an additional disk, so pool tank=driveletter x is ok. It becomes complicated if you enable a driveletter to a sub filesysten like 'tank/data=driveletter y' with the effect that the ZFS view of things is different to the Windows view.

For an experienced ZFS user this is no problem, an average Windows user may become very confused, Best is to avoid such settings. Best is to use the Unix way of mounting a pool tank -> /tank (c:\tank). Mounting a whole pool under a driveletter: is also ok. Avoid any settings where daughter/sub filesystems are not mounted below the pool. The "Nexenta way" seems also ok as they mount all pools under /volumes.

[lundman]

The default is to assign a driveletter to the pool, and all sub-datasets go inside that. The driveletter has a default setting of "poolonly" with that in mind. So if users do nothing, this is what will happen.

Mounting inside C:/ actually comes with a bunch of issues. If C: is case-insensitive, and your pool is not, Windows has difficulty crossing into C:/pool with different "mount" properties like that. A user can make it so the pool goes into C: if they really want to, with the assumption that they then also know to match the two filesystem options.

[guenther-alka]

On Windows and OSX default is case insensitive.
For (SMB) filer use, case insensitive is also the default. This is why case insensitive is always my pool default even on Unix.
Mixed can be an option for Unix/Linux and local apps but I would avoid case sensitive in general if you want to use it as a SMB filer.

Different properties of (daughter) filesystems than on parent or pool level is always a problem to consider. Beside case settings, this are mainly character sets, ex utf8, normalization and nbmand settings that should be the same across pool and nested filesystems.

edit: current napp-it nightly (feb 03) supports to modify atime, nbmand or readonly

[guenther-alka]

A Windows SMB filer with ZFS, why and when?

There was a discussion about this and the answer was:
Windows SMB direct with RDMA can be a powerful option with high performance needs ex for (multiuser) 4k+ video editing and large (Petabyte) storage that is more secure, feature rich and faster than ntfs/ReFS.

When it comes to performance over security:
https://www.borncity.com/blog/2023/02/14/windows-10-11-grottige-netzwerktransfer-leistung-hohe-windows-11-cpu-last-teil-1/

https://www.borncity.com/blog/2023/02/14/windows-11-netzwerktransfer-leistung-und-cpu-last-optimieren-teil-2/

(in German, use Chrome to translate on the fly)

SMB Performance tuning

https://github.com/MicrosoftDocs/windowsserverdocs/blob/main/WindowsServerDocs/administration/performance-tuning/role/file-server/smb-file-server.md

[guenther-alka]

Some impressions with SMB direct/RDMA and 100G (use Chrome to translate on the fly)
https://www.hardwareluxx.de/community/threads/microsoft-hyper-v-stammtisch.1114189/page-22#post-30262245

[guenther-alka]

I have a quite curious problem.

I updated Win11 on ESXi from 99-3 to 99-5. and rebooted. After that i wanted to import my pools via zpool import in the web-gui but no pool was found. I saw only disk messages like

error: path '\\?\scsi#disk&ven_vmware&prod_virtual_disk#4&1656f219&0&000900#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}' and '\\?\PhysicalDrive8' read partitions ok 2 gpt 0: type 892ecfda70 off 0x100000 len 0x7f600000 gpt 1: type 892ecfda70 off 0x7f700000 len 0x800000 asking libefi to read label EFI read OK, max partitions 9 part 0: offset 800: len 3fb000: tag: 4 name: 'zfs-0000238000007c83' part 8: offset 3fb800: len 4000: tag: b name: '' path '\\?\scsi#disk&ven_vmware&prod_virtual_disk#4&1656f219&0&000200#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}' and '\\?\PhysicalDrive2' read partitions ok 2 gpt 0: type 892ecfda70 off 0x100000 len 0x13f600000 gpt 1: type 892ecfda70 off 0x13f700000 len 0x800000 asking libefi to read label .....

After that i tried the zpool import locally at console. I saw the same disk messages but after that importable pools were listed and I was able to import the pools via cli.

Only difference between the local zpool import and the import via napp-it is that that napp-it uses a client server architecture with a background socketserver that executes the commands as admin via ipc::run and powershell (gives back stdout + stderr and respects a timeout) while locally I call zpool directly. So not really a bug in Open-ZFS but a weird behaviour.

[lundman]

I'm not sure on that myself, not an area that I am familiar with

[guenther-alka]

I have not seen this on the 99-3 release but there is a workaround for now

I have added a remark in the web-gui on Pool > Import when no importable pools are shown
-> open a terminal (admin ) and search for importable pools via 'zpool import' or 'zpool import -D'
and import with 'zpool import -f poolname'

[guenther-alka]

Question: Is it possible to access ZFS Snaps on Windows via the "previous versions" tab?
As this is allows access to single file or folder versions, it would be superiour over only making snaps as a whole filesystem version visible.

[sskras]

I guess in such case the project will need to implement additional VSS Provider:
https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service#shadow-copy-providers

[guenther-alka]

Not essential but ZFS snaps = VSS locally would be a huge win and the selling point for ZFS on Windows.

Another question:
In a benchmark, cpu load on ZFS was >80% vs 4% under ntfs,
performance of ZFS was 50-70% of ntfs.

Is this due debug code or is there another reason?
https://www.hardwareluxx.de/community/threads/napp-it-f%C3%BCr-zfs-on-windows.1349039/page-2#post-30272001
you can translate on the fly via chrome

[lundman]

I peeked at the API, so I know it exists - so it is on todo, to at least see if we can do something with it.

[sskras]

The only 3rd party FS implementation on Windows I managed to find that provides VSCopies is VxFS:

• How to register or unregister the Veritas Volume Shadow Copy Service provider with the Microsoft Volume Shadow Copy Service in Veritas Storage Foundation for Windows

• Frequently asked questions on Microsoft Volume Shadow Copy Services (VSS) and NetBackup Snapshot Client

Almost no matches for "Volume Shadow Copy Service" and "WinBTRFS" on Google.
I bet that would take a lot of effort.

[guenther-alka]

Another point
Windows detects the ZFS pool as physical 512B disk (512n) and not as 512e (4k)
Can this be the reason for the low performance?

https://www.hardwareluxx.de/community/threads/napp-it-f%C3%BCr-zfs-on-windows.1349039/page-3#post-30272238)

[lundman]

Use ashift=12 to overrule it?

[guenther-alka]

I expect then the following behaviour.

As Windows thinks, the ZFS "disk" is 512B physical/logical, it splits any data it writes in 512B chunks. As ZFS is aware of the real 4K physical disk structure, it must write every of these 512B chunks as a 4K block what means ZFS must write 8x the amount of data to the pool that Windows writes to the ZFS "disk".

This would declare the bad ZFS performance under Windows compared to the ntfs disk that reports the correct 512E value (512B logical/4K physical).

ZFS should report the "real" physical/logical pool structure or at least 512B logical/4k physical to go around this problem in most cases as ashift is not a pool but a vdev value and future NVMe may go to 8k/ashift 13. ZFS Pools with real 512B locical/physical are no longer to be expected as you cannot buy 512B native disks nowadays.

btw
I have seen that Open-ZFS on Windows can be tuned by regedt32 and
Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\OpenZFS\

There is an entry zfs_vdev_min_auto_ashift (undefined/invalid). Can this be used to modify or set ashift for the Windows view of a pool?

I have also asked the Gemini KI how to modify sectorsize for a ZFS pool in Windows
Answer:

goto: HKEY_LOCAL_MACHINE\SOFTWARE\OpenZFS\Configuration (not there)
add: Dword 32bit: PreferredSectorSize and set to 4096
restart

Entry should be valid for newly created pools
I know, KI answers are often not true and I hesitate to change registry without knowing the effect

napp-it on Windows Feb 12b first beta
Raid-Z expansion and draid support
jobs now working (scrub, snap and other)
new menu System with kstat and registry infos

[besterino]

@lundman: Pool was set at ashift=12. I am the one tinkering with ZFSonWindows in the other Forum/posts and set up a system with two identical NVME SSD to compare/ identify anomalies against native Windows filesystems. Happy to try whatever you throw at me.

[besterino]

For what it's worth: I recreated the process and made a screenshot how the fsutil reporting changes from a NTFS-volume compared to the ZFS dataset:

As it's in German, let me know if you need anything translated. But I hope it is clear enough that with an NTFS volume windows recognizes the logical sectors of 512 Bytes as well as the physical 4k sectors, whereas with a ZFS volume on the same SSD windows treats it as 512 "native".

[lundman]

I do believe I hardcoded 512 in one of the billion ways you can ask the filesystem, it just needs updating to report the actual value if that is the case..

[besterino]

If I may, where did you "hardcode" that? If it's in the registry, I'm happy to play around with different values and report results here. If it's in some "real" code, I might meet my limits though...

[guenther-alka]

I supppose a 512e default setting with 512/4096 can improve ZFS performance.
Maybe a overrule option in registry would be future proof for a possible ashift 13.

[besterino]

Disclaimer: I am in no way a programmer / coder, just a (maybe overly) excited enthusiast.

As to hardcoding the sector information, maybe would be an alternative to query the relevant information from the disk/windows? Something like IOCTL_DISK_GET_DRIVE_GEOMETRY_EX (source)?

As for scripting, so far I only found the "get-disk" powershell cmdlet which works on any & all drives, especially those not having a drive letter and not initialized by windows (both of which fsutil above unfortunately requires). Example to get relevant information for all disks:

get-disk | format-list -property Number,SerialNumber,LogicalSectorSize,PhysicalSectorSize

If all this is obvious, let me know. I am just trying to understand things a bit better.

[sskras]

Strangely enough my SATA SSD device reports 512-byte long physical sectors (while NVMe SSD disk reports 4096 bytes):

[steel3d]

I have the same mix of 512 and 4k. Only the crucial NVME reports 4k. Samsung 970 NVME reports 512. But perf is not great on the SSDs with 512 anyway.

`PS C:\Users\Steve> get-disk | format-list

UniqueId : eui.6479A787C0000410
Number : 0
Path : \?\scsi#disk&ven_nvme&prod_ct4000p3pssd8#5&22246615&0&000000#{53f56307-b6bf-11d0-94f2-00a0c91efb8
b}
Manufacturer :
Model : CT4000P3PSSD8
SerialNumber : 6479_A787_C000_0410.
Size : 3.64 TB
AllocatedSize : 4000787030016
LogicalSectorSize : 512
PhysicalSectorSize : 4096
NumberOfPartitions : 2
PartitionStyle : GPT
IsReadOnly : False
IsSystem : False
IsBoot : False

UniqueId : eui.6479A787C000049D
Number : 1
Path : \?\scsi#disk&ven_nvme&prod_ct4000p3pssd8#5&36f06839&0&000000#{53f56307-b6bf-11d0-94f2-00a0c91efb8
b}
Manufacturer :
Model : CT4000P3PSSD8
SerialNumber : 6479_A787_C000_049D.
Size : 3.64 TB
AllocatedSize : 4000785981440
LogicalSectorSize : 512
PhysicalSectorSize : 4096
NumberOfPartitions : 2
PartitionStyle : GPT
IsReadOnly : False
IsSystem : False
IsBoot : False

UniqueId : EUI.a63fb58154382500
Number : 2
Path : \?\scsi#disk&ven_nvme&prod_samsung_ssd_970#5&6ee56e9&0&000000#{53f56307-b6bf-11d0-94f2-00a0c91efb
8b}
Manufacturer : NVMe
Model : Samsung SSD 970
SerialNumber : 0025_3854_81B5_3FA6.
Size : 476.94 GB
AllocatedSize : 512109142016
LogicalSectorSize : 512
PhysicalSectorSize : 512
NumberOfPartitions : 5
PartitionStyle : GPT
IsReadOnly : False
IsSystem : True
IsBoot : True

UniqueId : 500253F833545F06
Number : 3
Path : \?\scsi#disk&ven_samsung&prod_ssd_870_evo_2tb#5&1a6bbaae&1&020000#{53f56307-b6bf-11d0-94f2-00a0c9
1efb8b}
Manufacturer :
Model : Samsung SSD 870 EVO 2TB
SerialNumber : S6PNNS0W509171J
Size : 1.82 TB
AllocatedSize : 2000398934016
LogicalSectorSize : 512
PhysicalSectorSize : 512
NumberOfPartitions : 1
PartitionStyle : MBR
IsReadOnly : False
IsSystem : False
IsBoot : False

UniqueId : 500253F8314405B3
Number : 4
Path : \?\scsi#disk&ven_samsung&prod_ssd_870_evo_2tb#5&1a6bbaae&1&030000#{53f56307-b6bf-11d0-94f2-00a0c9
1efb8b}
Manufacturer :
Model : Samsung SSD 870 EVO 2TB
SerialNumber : S620NJ0R410042Y
Size : 1.82 TB
AllocatedSize : 2000398934016
LogicalSectorSize : 512
PhysicalSectorSize : 512
NumberOfPartitions : 2
PartitionStyle : GPT
IsReadOnly : False
IsSystem : False
IsBoot : False

`

[besterino]

My Samsung SM961 reports 512/4k:

PS C:> get-disk | format-list

UniqueId : eui.002538CC610028BC
Number : 0
Path : \?\scsi#disk&ven_nvme&prod_samsung_mzvpw256#7&2fb25a1f&0&000000#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}
Manufacturer :
Model : SAMSUNG MZVPW256HEGL-00000
SerialNumber : 0025_38CC_6100_28BC.
Size : 238.47 GB
AllocatedSize : 256059465728
LogicalSectorSize : 512
PhysicalSectorSize : 4096
NumberOfPartitions : 4
PartitionStyle : GPT
IsReadOnly : False
IsSystem : True
IsBoot : True

...as do a Samsung P9A1 and a WD SN850X. My Solaris Comstar iSCSI-Share reports 512/512 though... ;)

[guenther-alka]

A problem may occur if Windows thinks ZFS is a 512B native disk while ZFS is using 4K/ashift 12 what can mean that a 512B write on Windows results in a 4K write in ZFS.

Unless there is an option to switch the pool on Windows from 512N (512B logical/physical) to 512E (512B logical/4K physical) one can only asume that this is the reason for the lower ZFS performance.

[steel3d]

I tried ashift=9 on my 512B logical/physical SSD and the performance is unchanged.

[guenther-alka]

Sounds not too promising about a better ZFS performance with a 4k disk setting now.
If there is no change we need to wait for the small steps to improve performance when ZFS on Windows reaches a stable state.

[lundman]

You mentioned IOCTL_DISK_GET_DRIVE_GEOMETRY_EX which is here
https://github.com/openzfsonwindows/openzfs/blob/windows/module/os/windows/zfs/zfs_vnops_windows_lib.c#L5517-L5568
and you can see I cheated what I wrote the code there with hardcoded 512. (Same with ioctl_disk_get_drive_geometry)

heh can't find a single place where I didn't cheat.

[besterino]

Thank you! Will try to have a look tomorrow.

[besterino]

Oh my, this is beyond my ability to understand. I don't even know what exactly I am looking at. As I don't want to keep you away from the more important stuff, I'll wait for the next update... ;)

[guenther-alka]

Napp-it for ZFS on Windows now supports groups to manage several Windows ZFS servers. This will be extended later to non-Windows servers like OmniOS or Linux but BSD, OSX or SmartOS are also possible options.

Download (unzip and replace /xampp/[nappit]/[data] ([names] can be modified). Xampp portable must be already downloaded
To update to newest, download

https://napp-it.org/doc/downloads/napp-it_nightly.zip

unzip folder [napp-it]

• stop Apache and background services (cmd windows)

• replace c.\xampp[napp-it] with newer version.

newest infos:
https://www.napp-it.org/doc/downloads/napp-it on windows.pdf