ZFS out-of-order writes after System crash. ZIL partial txg replay ?

[shantanukshire]

Problem: User application issues pwrites to certain zvol offsets. System crash occurs while application IO is ongoing. After the system is back online, zvol data is inconsistent (not in order). This issue occurs specifically for System crash during an in-flight flush transaction.

Here is an example of application writes/fsync issued to zvolume.

/**
Crash occurs before final fsync() return
Txg states during system crash. 29 (sync), 30 (open)
The offset in concern is 0x1300000000
*/

Writes to ZVOL offset=0x1300000000, data=A00 [txg=29]
Writes to ZVOL offset=0x1342750000, data=(some random data) [txg=29]
Writes to ZVOL offset=0x1342760000, data=(some random data) [txg=29]
Writes to ZVOL offset=0x1342770000, data=(some random data) [txg=29]
Writes to ZVOL offset=0x1300000000, data=B00 [txg=29]
Writes to ZVOL offset=0x1300000000, data=BA0 [txg=29]
Writes to ZVOL offset=0x1342fe0000, data=(some random data) [txg=29]
Writes to ZVOL offset=0x1342ff0000, data=(some random data) [txg=30]
Writes to ZVOL offset=0x1343000000, data=(some random data) [txg=30]
Writes to ZVOL offset=0x1300000000, data=BB0 [txg=30]
Writes to ZVOL offset=0x1300000000, data=BBA [txg=30]
fsync() call to ZVOL 

Post crash startup ZIL is replayed. However, since flush was incomplete we basically see replay of a partial txg. I get below offset data after parsing ZIL content.

offset=0x1300000000, data=BBA (txg=29)
offset=0x1342750000, data=(some random data)
offset=0x1342760000, data=(some random data)
offset=0x1342770000, data=(some random data)
offset=0x1300000000, data=BBA (txg=29)
/* No records in ZIL after this point. Not all writes able go in due to crash */

My understanding is that post flush call, ZFS starts writing down all incomplete txgs to ZIL (in above scenario it is txg 29 and 30).

The data "BBA" should be written only after previous writes i.e. 0x1342fe0000, 0x1342ff0000, 0x1343000000, but this is not true! The data content for the common block (0x1300000000) seems to be taken from Open txg. This is a problem with ordering.

Note that if we disable ZIL replay then issue does not occur. So there is certainly a problem with partial txg replay.
I'm new to ZFS internals so have a few question.

• Is this intended behaviour? Why does ZFS replay partial txg? (aren't txg expected to be atomic)
• Can this issue be prevented within ZFS?
• Is it good idea to altogether disable ZIL txg replay? Can ZIL replay be disabled only for single pool?

Note that we don't do sync=always. The fsync() calls are used sparingly for performance reasons
(The ZFS version in context is 0.7. Linux Kernel: 3.10.0-1062.18.1.el7.x86_64)