Meta List regarding satisfaction of Wipe Standards

[fthobe]

WARNING Rolling Release Base, Changes / Additions might occur

Current State of Issue

Missing Components in this issue

• List all Standards (work in progress) and compare
  Full document:
• Verify against current PDF
• Verify against current csv
• Verify tech against wipe standards

Data Wipe Certification According to Standard

The current certification provides significant value, but does not contain all mandatory and / or optional fields to certify the destruction of the disk contained data. This issue outlines eventual shortcomings.

Applies to following standards

The following table illustrates the requirements for the documentation of a sanitisation according to various standards.
Currently documented are:

1. US: NIST SP 800-88 Rev. 1 requires a form similar to the appendix of the linked standard.

2. US: DoD I 8500.01 (refers to NIST SP 800-88) DOD M 4160.21 Vol 4 requires form 2500 similar to the document here.

3. US: DoD 5220.22-M requires a very reduced form

4. Canada: ITSP 40006: see US: NIST SP 800-88 Rev. 1

5. Germany: CON.6

Particular Requirements

Different Tool for Verification

Following standards require a different tool for verification:

1. NIST SP 800-88 Rev. 1
2. ITSP 40006 according to NIST SP 800-88

Control of Disk Size

Following Standards require to mathematically compare disk size before cancellation and after verification:

1. ITSP 40006

Validation of Tool itself

Following standards require to validate if the tool used is tamper free (eg checksum of binary)

1. ITSP 40006

Tool needs to be bootable

Following standards require the applications to run from a bootable device:

1. ITSP 40006

Following Standards require to Destroy Disks with Bad Sectors

Following standards require to verify bad sectors for successful erasure or otherwise return failed:

1. ITSP 40006

Overwrite with Random Values

Following Standards require PNRG Streams to overwrite

1. Germany: CON.6

DOES NOT APPLY TO FOLLOWING STANDARDS

1. NSA/CSS Policy Manual 9-12 as it is a destruction only standard

Additional Sources

1. Canada: From RCMP TSSIT OPS-II to ITSP.40.006. Canada’s Data Sanitization Standards. Explained.

[Knogle]

I've gone through in depth, looks like a nice goal.
I also have specific use cases for that.

[Firminator]

Following Standards require to Destroy Disks with Bad Sectors

Following standards require to verify bad sectors for successful erasure or otherwise return failed:

[ITSP 40006](https://www.cyber.gc.ca/en/guidance/it-media-sanitization-itsp40006#fnb5)

This is a real-world issue right there. I encountered many, many drives with bad sectors/blocks which then fail to wipe properly and just seem to abort as there is no logic to jump over the bad blocks.
Looking at https://github.com/users/PartialVolume/projects/1 I see this is a backlogged feature under point 34 and 35 with an XL amount of coding involved. I remember we discussed this briefly and PartialVolume came up with a preliminary logic how to implement this. Any chance to increase priority of 34 and 35 - dealing with BadBlocks - possibly even over the SecureErase implementation that you mentioned elsewhere which will be the next big thing you want to work on? Just checking and mentioning this here, no pressure.

[fthobe]

Hey guys,

Some notes:

1. @Knogle it's not done yet, I will finally have some time this weekend
2. @Firminator actually no, if a report considers the disk not certifiable it's not certifiable, work done. The much bigger issue is the integrity of the information if a disk has broken sectors or not.

[Firminator]

fthobe, use cases are manyfold. Not everyone needs a cert at all times. Think for example server drives 'failing' due to SMART errors (usually hitting a treshold of bad blocks) and needing to be removed from production and moved into the re-use/recycling process. You still would want the data un the healthy blocks wiped, but you don't expect the drive to be certified sanitized on every block.
Or a situation where a user wants to ensure that all data is removed before throwing away a drive. In the past nwipe fails at the first bad block and leaves the remaining blocks behind the bad block untouched. Error -1 or something similar will then be shown in the log.

[fthobe]

I do absolutely agree, a certification is not an obstacle to an incomplete / partial wipe, it just can not be certified afterwards.

I do not think a softfail (for example logging non wipe able block and continuing to write while logging the bad blocks with a certificate stating it) wrong, but it shouldn't be in this issue :)

[PartialVolume]

Following Standards require to Destroy Disks with Bad Sectors
Following standards require to verify bad sectors for successful erasure or otherwise return failed:

[ITSP 40006](https://www.cyber.gc.ca/en/guidance/it-media-sanitization-itsp40006#fnb5)

This is a real-world issue right there. I encountered many, many drives with bad sectors/blocks which then fail to wipe properly and just seem to abort as there is no logic to jump over the bad blocks. Looking at https://github.com/users/PartialVolume/projects/1 I see this is a backlogged feature under point 34 and 35 with an XL amount of coding involved. I remember we discussed this briefly and PartialVolume came up with a preliminary logic how to implement this. Any chance to increase priority of 34 and 35 - dealing with BadBlocks - possibly even over the SecureErase implementation that you mentioned elsewhere which will be the next big thing you want to work on? Just checking and mentioning this here, no pressure.

@Firminator I've bumped direct I/O and forward & reverse no abort on block errors up the priority list. Items 21-23 in Priority List Once I've got direct I/O working reliably (I had already started a branch for direct I/O, but got side tracked) I can then work on the code to skip the blocks. This whole feature requires direct I/O because we not only need to retain the speed of cached I/O by using our own 120,000 block buffer (for each wipe in progress) but when an I/O error occurs on the 120k block write we then need to identify the failed block, so we start sending the 120k block again but this time in 512 or 4096 block writes, instead of the default 120k block write which of course will slow transfer speed, until we get an arbitary 10 contiguous blocks with no I/O error then we expand the write size to increase write speed as rapidly as possible. The bad blocks will be logged in the nwipe.log. The format being:

[2024/10/17 11:25:16]   error: bad blocks found 10444568-10445477

which will show the range of contiguous bad blocks.

None of the above would be possible if we continued to rely on the kernel to cache our writes.

[fthobe]

Hey, I am closing this up tomorrow, I added some standards and will tomorrow night publish a big update to this post. Some take aways: DoD standards are dead, literally. All American documents now point to NIST.

Regional standards are either in the process of being revised or are either pointing to NIST or are a version of enhanced NIST.

I have the feeling that some things might not be achievable without added features:

• there might be requirements for personal information storage that require consent in the European Union (Sanitation Manager / Reviewer)
• Binary Checksum check for Canada needs a strategy to verify the integrity of the application even when coming from read only media
• Bad blocks need to be protocolled and for some standards cause the certificate to report a failure of erasure if present or reaching thresholds (which is really difficult to protocol given that disk controllers might shield that information
• Some standards can only be realized from non modifiable media (iso / pxe read only) which makes documentation for pxe deploy an aspect thinking of WDS and Fogg / iPXE as most used solution
• It seems to be common understanding that while verification requires in some cases an additional software for verification, that doesn't necessarily mean that this software does not come from the same media (eg nwipe as front end for nwipe erase and nwipe verify for example)

I would suggest to finish up the table and have a larger discussion to validate what @PartialVolume and @martijnvanbrummelen feel that can be achieved and where and maybe a more extensive community support is needed, I might be able to get some funding for some issues.

In any case the big issues are from what i see:

• bad sectors
• Intransparent disk behavior
• intransparent disk reporting of bad sectors

I feel that there's a big research component also regarding the compliance of drives that should include manufacturer / vendor support. I have good reach at Samsung through my work in Italy.

How do you people feel about that?

@Knogle @Firminator

[Firminator]

Looks like the new standards are tailored or were influenced by Blancco. I guess they can only be achived if you have funding for several full time developers. Way over the goals of what ShredOS and nwipe try to provide: a wiping program for the rest of us that have no need for a multi-thousand dollar solution. Enthusiats, privacy-minded people, one-off wiping demands, educational institutions, and non-profits.

[Knogle]

Usually disk manufacturers have some proprietary APIs to query info like the actual bad block out of their firmware.
But I think that's way beyond the scope of nwipe unfortunately and also out of reach for us.

[fthobe]

Looks like the new standards are tailored or were influenced by Blancco. I guess they can only be achived if you have funding for several full time developers. Way over the goals of what ShredOS and nwipe try to provide: a wiping program for the rest of us that have no need for a multi-thousand dollar solution. Enthusiats, privacy-minded people, one-off wiping demands, educational institutions, and non-profits.

@Knogle Actually no, it's not a private API problem, it's a disk response problem:

• all major manufacturer products to respond to SATA / SAS / NVME commands, the problem is that the reply is often not correct

Looks like the new standards are tailored or were influenced by Blancco. I guess they can only be achived if you have funding for several full time developers.

@Firminator actually NVME is definetly a step forward, given that it was an Open Source driven client implementation, situation got much better recently.

[fthobe]

Looks like the new standards are tailored or were influenced by Blancco. I guess they can only be achived if you have funding for several full time developers. Way over the goals of what ShredOS and nwipe try to provide: a wiping program for the rest of us that have no need for a multi-thousand dollar solution. Enthusiats, privacy-minded people, one-off wiping demands, educational institutions, and non-profits.

I think they literally just default to pay for failed destruction and destruct physically.