Skip to main content

Filesystem Journaling: Write-Ahead Logging

Follow a filesystem transaction from journal to checkpoint. Explore crash recovery, ext4 journal modes, and the limits of data durability.

The Problem: Crashes During Writes

Appending a block to a file changes more than its contents. The filesystem must also update the inode's size and block references, and mark the new block as allocated in a bitmap. These structures live in different places on disk. A power failure between their writes can leave them disagreeing about who owns a block.

A consistency checker can scan the filesystem to reconcile those structures, but the work grows with the filesystem. Journaling records a recoverable transaction so ordinary crash recovery can focus on recent updates. The goal is consistent filesystem structures; preserving every recent application write is a separate requirement.

The Journaling Solution

Write-ahead logging puts recovery information on stable storage before overwriting the corresponding home metadata. A complete transaction has a durable commit record. After that point, the filesystem can copy its logged updates to their home locations, a process called checkpointing.

If power fails during checkpointing, recovery replays the committed records. In the redo-journal model used here, records without a valid commit are ignored. They do not require undoing home metadata because those updates were not allowed to reach home before commit. OSTEP's crash-consistency chapter develops this ordering from the inode-and-bitmap problem.

How Journaling Works: Interactive Exploration

Follow an 8 KiB file growing to 12 KiB, using three 4 KiB blocks. In Crash Recovery, compare failures before commit, after commit, and during checkpointing. Watch the journal, inode, allocation bitmap and file payload separately.

What survives a crash?

Follow one append through the journal and its home locations. The durable commit determines what recovery can replay.

Step 1 / 6

Start with a consistent file

doc.txt occupies blocks 100 and 101. Append 4 KiB by reserving block 102 in memory.

Transaction journal

No active records

Inode after-image
No active record
Bitmap after-image
No active record
Commit record
No active commit

Home locations

doc.txt · 4 KiB blocks

Inode size
8 KiB
Inode block references
100101
Bitmap for block 102
free
Payload in block 102
New data not durable

No transaction has reached stable storage yet.

Authored illustration, not a disk trace. The transaction views model an ordered-mode append using 4 KiB blocks. Journal rows summarize metadata after-images; real JBD2 records also contain descriptors, checksums and revocation information.

Journal Modes: Safety vs Performance

The explorer's transaction uses ext4's ordered-mode rule. ext4 also offers full data journaling and writeback mode; the choice changes which bytes are recoverable and which writes commit must wait for. These names describe ext4 options, not a universal set of modes shared by all filesystems.

1. Journal Mode (Full Journaling)

With data=journal, file contents and metadata pass through the journal. Committed journaled contents can be replayed before their home copies are complete. The additional payload writes increase I/O traffic, but do not make arbitrary application operations atomic.

2. Ordered Mode (Default)

With data=ordered, metadata is journaled and associated file data must reach its home blocks before the metadata transaction commits. Data and journal I/O can overlap; the requirement is the dependency at commit, not one fixed serial schedule. File data itself is not available in the journal for replay, and in-place overwrites are not made atomic.

3. Writeback Mode

With data=writeback, metadata is still journaled, but file-data writeback need not finish before metadata commit. A crash can therefore leave unexpected or stale contents in recently written files. Metadata still obeys write-ahead ordering. See the ext4 journal format and ext4 mount options for the guarantees and tradeoffs.

The Transaction Lifecycle

The important milestones are different kinds of completion:

  1. Record: describe the metadata update in the journal.
  2. Flush prerequisites: make those records durable and satisfy the mode's data-ordering requirement.
  3. Commit: make a valid commit record durable, establishing that recovery may replay the transaction.
  4. Checkpoint: write the committed metadata to its home locations.
  5. Reclaim: after those home writes are durable, reuse the transaction's journal space.

A successful buffered write() is not the same as a durable journal commit. Applications use operations such as fsync() and must check their results. Persisting a newly created or renamed directory entry can also require syncing its containing directory. The Linux fsync(2) documentation explains that boundary.

Recovery After Crash

Recovery examines the active journal, validates its records, and replays complete transactions that still need checkpointing. Incomplete transactions are discarded. Replay must tolerate a home write having already completed before the crash; restoring the same logged metadata value again should have the same result.

That is why the explorer's partial-checkpoint case ends with the same inode and bitmap as its after-commit case. The before-commit case keeps the old file, even though new payload bytes may already exist in an unreferenced block.

Recovery work depends on the active log and storage performance. Journaling reduces the need for a full consistency scan after an ordinary crash; it does not repair every form of corruption or eliminate filesystem repair tools.

Journaling in Different Filesystems

ext4

ext4 normally uses JBD2 with an internal journal, and can use a separate journal device. Its journal size depends on filesystem configuration. Fast commits can log a smaller description of eligible metadata changes; the explorer illustrates the conventional transaction path rather than that optimization. See the JBD2 documentation.

XFS

XFS logs filesystem metadata using its own mix of physical records, logical updates and intent records. It is not ext4 running in data=ordered mode. An external XFS log is also distinct from an XFS realtime data device, as described in xfs(5). The XFS logging design explains its transactions and recovery machinery.

NTFS

NTFS uses a transaction log and checkpoint information to recover filesystem consistency. Its USN change journal serves applications tracking changes and is a separate mechanism. Microsoft's NTFS overview describes recovery and when repair may still be necessary.

Btrfs / ZFS

Copy-on-write filesystems build changed structures in new locations and then publish a new committed tree. That does not exclude logging: Btrfs has a tree log, and ZFS has an intent log for synchronous operations. Their recovery paths differ from the conventional metadata journal shown here. Continue with the Copy-on-Write mechanism for the tree-update model.

Performance Impact

Journal Placement

A separate journal device can reduce contention between log writes and other I/O, especially on seek-bound storage. The effect depends on the device, queueing, workload and durability requirements; an external log is not automatically faster. Recovery depends on the journal device as well as the main data device.

Before tuning, inspect the filesystem and effective mount options for the path you care about:

bash
# Inspect the filesystem containing the current directory.
findmnt -T . -o TARGET,SOURCE,FSTYPE,OPTIONS

Journal Size Tuning

More journal space can accommodate more outstanding metadata work before space pressure forces checkpointing. It does not directly change the periodic commit interval. The useful size depends on the workload; throughput and latency claims need measurements with the same durability settings.

For an ext4 filesystem, inspect the configured journal using its actual block-device path:

bash
# Replace this placeholder with the ext4 device identified above.
sudo tune2fs -l /dev/your-ext4-device

This lists configuration; it does not monitor journal wraps. dumpe2fs -h provides another read-only view of an ext4 device's superblock and journal configuration. The tune2fs(8) manual documents journal options and their constraints.

Best Practices

  • Choose a mode for its guarantees. Start from the filesystem's defaults and evaluate changes against a specific workload.
  • Request application durability explicitly. Use the appropriate sync operations, check errors, and follow the application's recovery protocol.
  • Keep storage ordering intact. Journal recovery assumes that the storage stack honors flush and ordering requirements.
  • Measure before tuning. Compare latency and throughput under equivalent durability settings.
  • Keep backups. A consistent filesystem can still contain unwanted changes or suffer device failure.

Journaling vs Copy-on-Write

AspectConventional journalingCopy-on-write trees
Update strategyLog changes, then update home locationsBuild new blocks, then publish a new tree
Extra writesJournal traffic; which bytes are duplicated depends on modeNew data and metadata paths; additional logging may apply
Crash recoveryReplay valid committed log recordsSelect committed roots and replay relevant logs
SnapshotsNot supplied by journaling itself; other layers may provide themShared tree blocks support snapshots, with later space costs
Application durabilityRequires the appropriate sync protocolRequires the appropriate sync protocol

Neither technique guarantees that every recent application write survives. The distinction is how filesystem updates become recoverable. See the Btrfs design and OpenZFS copy-on-write design for how tree updates and logging coexist.

If you found this explanation helpful, consider sharing it with others.

Mastodon