ZFS requires HBA IT mode and JBOD mode on RAID cards not supported

[johngrabner]

Many in the Truenas community say that JBOD mode on RAID cards will appear to work with ZFS, but it causes problems in the past and probably does not work now.

Are there test cases in ZFS or elsewhere that could validate the proper operation of JBOD mode on RAID cards?

Does anyone know if the issue of JBOD mode still exists?

Is there a compatibility matrix of cards/firmware and ZFS?
Specifically, I purchase a server with a 9361-8i, and this card claims to support
JBOD mode.

[quartsize]

Hi John! I was under the impression "IT mode" and "JBOD mode" were synonyms. It may help folks to answer this if you can pass along what the specific concerns were/are. You may also find informative this recent thread on zfs-discuss about potential problems in the absence of true pass-through.

[johngrabner]

I posted a question if JBOD mode on a RAID card 9361-8i would work with Truenas in the Truenas forum. Here is the link. They responded with generalities and statements that historically only HBA IT mode firmware was reliable and that the problem persists today with all RAID cards in JBOD mode. When I asked if they have test cases that fail (ie: specifics) , they offer none.

[youtube](Forbidden Arts of ZFS | committing the greatest sin & getting away with it [Hardware RAID with ZFS) claims you can even use ZFS with Raid 0 disk. Presumably, even less transparent passthrough than JBOD mode.

So I figured I would ask people who develop ZFS what they know about this raid card can't support ZFS even in JBOD mode.
If there is something wrong with JBOD mode on these RAID cards, I would love to raise a trouble ticket with Broadcom. But I can't do this with general statements.

Can you offer some light on this?

[amotin]

Plain HBA or what is called "IT mode" does not do anything other than delivering commands and data to the disks. There is no configuration, no internal state, no data processing, no caching, there is nothing. It means you can remove any failed HBA and replace it with any other any moment, or move disks between different HBAs without thinking. This is the preferred configuration for ZFS since it provides it full control over the performance, reliability and error recovery. For TrueNAS it also provides full management abilities, since all disks have their original identification, support SMART, etc.

RAIDs in JBOD mode may vary between vendors and models. Some RAIDs, like LSI HBA with IR firmware may be indistinguishable from plain HBAs, just their firmware has some unused code, that is not critical. Some others RAIDs may automatically convert all unconfigured disks into some forms of single-disk RAID0, that may be OK for ZFS, but may create problems for TrueNAS management, since some RAIDs do not pass through original disk identification, and TrueNAS requires unique disk serial numbers. Worst case of JBOD support requires explicit manual RAID0 configuration with some external tools, that makes hot-swap problematic and controllers not easily interchangeable.

Worst case are pure RAIDs, where you have to configure some real RAID volume(s). In that case you loose all control over what is going of with your data above what RAID can provide you, ZFS can not properly schedule I/O or do error recovery, you may suffer from RAID5 write holes, etc. This is absolutely no go for ZFS or TrueNAS.

[johngrabner]

Thank you.

How can one determine where in the spectrum a RIAD card in JBOD mode sits?
I was hoping for something like, run xxx if you see serial numbers, then xxx is ok.

[amotin]

Put in the card, connect the disks, without any configuration boot to to OS/TrueNAS and see whether you see all the disks as raw disk devices with model names, serial numbers, etc.

[johngrabner]

Thank you,

For linux, the command "lsblk --output SERIAL,MODEL" will list the serial number and model. A raid 0 will probably have a different model, maybe the same serial number or maybe a different serial number. Correct?

In the event disks are passed as Raid 0, do you know how Truenas is dependent on the serial number?
I asked because on my existing truenas using HBA in IT mode, I purchased the cheapest no-name disks, and Truenas reported odd serial numbers on occasion, but everything appeared to work from a ZFS perspective.

What about functionality like Scrub: would this care about Raid 0 vs native disk? I assume this is ZFS functionality.

The SMART GUI of truenas takes disk by "sbd, sbg, sbd". I take it this is nothing to do with ZFS.

Do you know if anyone maintains a list of HBA/Raid cards and firmware numbers that are known to work with ZFS?