Data Integrity

fsync is often assumed. This post uncovers those gaps and offers practical safeguards.

Write-ahead logging (WAL) writes changes to a durable log before modifying the database, allowing recovery after crashes and eliminating corruption.

Introduction In the world of storage systems, data integrity is a non‑negotiable requirement. A single corrupted byte can cascade into file system corruption, application crashes, or even data loss. While traditional journaling filesystems protect against power failures and crashes by replaying a write‑ahead log (the journal), they often assume the journal itself is trustworthy. In practice, hardware faults, memory errors, or transmission glitches can corrupt journal entries before they are applied to the main file system structures. ...

Introduction In the digital age, the reliability of data transmission, storage, and processing is taken for granted—until it isn’t. A single corrupted byte can render a downloaded file unusable, cause a network packet to be dropped, or silently introduce bugs into a software build. The unsung hero that helps detect (and sometimes correct) such errors is the checksum. A checksum is a compact, deterministic value derived from a larger body of data. By recomputing the checksum at the destination and comparing it to the sender’s original value, we can quickly verify whether the data has been altered. While the concept is deceptively simple, the world of checksums is surprisingly rich: from elementary parity bits used in early telegraphy to sophisticated cyclic redundancy checks (CRCs) embedded in Ethernet frames, and up to cryptographic hash functions that underpin blockchain integrity. ...