Filesystem Data Integrity: Detecting Silent Corruption

The silent corruption problem

Ordinary file-data reads on ext4 and XFS generally have no filesystem data checksum to compare against. If a disk reports success while returning corrupted bytes, those bytes can reach the application. Metadata checksums protect a different part of the filesystem; they do not verify ordinary file contents.

Corruption happens more often than you'd expect, and most of it never trips a SMART warning:

• Bit rot. Cosmic rays, magnetic decay, and NAND charge leak flip individual bits on stored media over months to years.
• Firmware bugs. RAID controllers and SSDs occasionally return wrong data because of bugs in their write coalescing or wear-leveling code paths.
• Misdirected writes. The controller writes data to the right LBA at the wrong moment, overwriting an unrelated block. The drive reports success.
• Phantom writes. The drive reports success but never actually persists the block — a subsequent read returns stale data from cache or the previous version on disk.
• DMA / memory errors. Bits flip in DRAM during transfer between CPU, controller, and disk. Without ECC RAM, you have no way to detect it.

The worst part is the silence. The disk returns success, the filesystem sees no error, the operating system has no signal. Your data is corrupted and nobody knows until you read it back months later and find a photo with a colored band across it, or a database that refuses to recover.

The checksum solution

ZFS and Btrfs can protect file data by recording a checksum when a block is written and verifying the bytes when it is read. The algorithm can be a fast non-cryptographic checksum or a cryptographic hash. These examples use real SHA-256 digests to make the comparison visible.

Flip one bit in block 0, then step through the read. Compare the result with checksum verification enabled and disabled:

The expected checksum needs its own protected metadata. ZFS stores a child's checksum in its parent block pointer. Btrfs stores file-data checksums in a dedicated checksum tree. Both separate the expected value from the bytes being checked, but their on-disk structures differ.

Why the tree shape matters

A checksum tree lets a reader verify the metadata that supplies a block's expected checksum. Starting from a trusted root, the reader checks the metadata along the chosen path, then checks the leaf bytes before using them. A bit flip changes disk data, not the stored parent checksums or root. Reading the root alone cannot reveal corruption in an unread leaf.

Block 2 starts with a bit flip. Read block 0 first, then select block 2: only the selected path is verified. Toggle bit flips on any of the four blocks to explore the difference between damaged and checked data:

The figure is a small teaching Merkle tree, not a replica of either filesystem's format. Production block sizes, metadata records, and checksum algorithms differ. OpenZFS's default checksum=on currently selects Fletcher4; SHA-256 is an available option, rather than the default.

This is what makes the integrity guarantee end-to-end. The filesystem proves that the bytes returned to the application are the bytes that were originally written, not bytes that happened to read back from the same sectors.

Self-healing with redundancy

Detection is half the work. With a mirror or a RAID-Z group, a checksum-aware filesystem can detect corruption and also repair it, provided a valid redundant copy or reconstruction is available. Follow the same block through a mirror read, then try two damaged copies or no mirror:

Three properties make this work that traditional RAID can't match:

1. The filesystem checks each candidate. When mirrored copies disagree, the expected checksum helps identify a valid copy. Redundancy alone does not establish which bytes are correct.
2. Recovery can serve the read. When an alternate copy verifies, the application receives its bytes and the damaged copy can be repaired. If no copy verifies, the read fails instead of returning corrupted bytes.
3. Repair results are observable. Error and repair counters help operators investigate damaged storage. The figure keeps disk A marked damaged until the verified rewrite occurs.

Scrubbing: proactive detection

Corruption that's never read stays hidden. A normal ZFS scrub examines the pool's referenced data and verifies block checksums. Redundancy allows damage to be repaired when a valid recovery source exists; unrecoverable errors are reported.

Ordinary reads below touch only the hot blocks B0 and B1. Switch to a scrub to discover the bit flip in cold block B2, then change the recovery scenario:

zpool scrub tank
zpool status tank

Cadence depends on the data:

• Critical or archival data: weekly. The window where both copies could rot independently has to be smaller than the rate at which you discover and replace a failing drive.
• General-purpose pools: monthly. The default on most ZFS-on-Linux installs.
• Mostly-read workloads: match scrub cadence to the typical re-read interval — there is no point scrubbing a block that the workload already reads daily.

The goal is to read every block more often than the median time between independent corruption events on two different disks. For consumer SATA drives that's roughly monthly. For enterprise SSDs the safe interval can stretch to quarterly.

Filesystem comparison

The cost of integrity

Checksums aren't free, but the overhead is small enough that on modern hardware you measure it in noise:

• CPU. 1–5 % for SHA-256 on a busy server; closer to noise with hardware-accelerated CRC32C or xxhash. On any CPU with SHA-NI or ARMv8 crypto extensions the cost disappears into a single cycle per byte.
• Space. ~0.1–0.5 % for checksum storage, depending on block size and which algorithm is configured.
• Latency. Hidden by the disk I/O time. Verification happens in parallel with the next block fetch.

For most workloads you won't notice. The narrow exceptions are high-frequency trading and write-amplification-sensitive databases, where you can leave metadata checksummed and disable data checksums on specific datasets.

Why dm-integrity isn't quite enough

On ext4 or XFS you can stack dm-integrity underneath to get block-level checksums without changing filesystems. It works, but it isn't equivalent:

• dm-integrity doesn't know which copy of a mirror is good. It will report an error on a corrupted read, but it can't pull a clean copy from a sibling device because mirroring lives in mdraid above it. You still lose the read.
• Each layer protects its own bytes, not the data path. Memory errors during DMA between layers go undetected.
• There's no scrub of the upper layer's data, only of the integrity tag the lower layer wrote.

dm-integrity is a real improvement over bare ext4, but it doesn't reach the end-to-end guarantee a checksum-aware filesystem gives.

When integrity matters most

The critical use cases are the ones where you cannot recreate the bytes:

• Long-term archival storage — family photos, scanned documents, backups that won't be read for years.
• Databases and financial records — silently flipped bits in a ledger or a column store are catastrophic.
• Scientific data — instrument readings, simulation outputs, anything where reproducibility costs serious compute.
• Source-of-truth caches — once a corrupted block is read into the cache, the corruption can propagate to every consumer until eviction.

If you're running ext4 or XFS without dm-integrity or filesystem-level checksums, you are trusting cosmic rays, firmware bugs, and disk aging never to corrupt your data. On a long enough timeline, they will.

Practical recommendations

1. New storage systems should be ZFS or Btrfs (mirror only, not parity-raid) with weekly scrubs. The marginal operational cost is small.
2. Existing ext4 or XFS that holds anything you can't recreate should be sandwiched with dm-integrity plus mdraid mirroring. The result is weaker than ZFS but vastly stronger than the default.
3. All servers should use ECC RAM. Without it, a memory error during DMA can corrupt every read your filesystem checksum would otherwise have protected.
4. Scrub before you back up. A backup of a corrupted block silently propagates the corruption into your archival copies.
5. Track scrub error counts. A drive that starts repairing more blocks per scrub than its baseline is a leading indicator of impending failure — replace it before it fails outright.

Related concepts