Corrupted Block Detector via Rolling Checksums
Storage drives suffer silent data corruption from bit rot and cosmic rays. Computing block-level Adler-32 or CRC checksums divides large disk extents into fixed chunks, allowing automated scrubs to identify exact corrupted disk block numbers without rewriting entire files.
import zlib
# 4KB block checksumming:
block_checksum = zlib.adler32(chunk)
# Corrupted Block Detector via Rolling Checksums When a magnetic disk or NAND flash cell experiences silent data corruption (bit rot), the file system may continue serving damaged data without throwing an OS error. If you only verify a single SHA-256 hash for the whole 50 GB database file, you know it is corrupted—but you have no idea *which* 4 KB sector broke. 🛑⚡ Storage architectures like ZFS and Btrfs divide storage pools into fixed-size **Blocks** (e.g. 64 KB). By maintaining a tree of block-level checksums: 1. Periodic background **Scrubbing** jobs read each block and compare its computed checksum against the catalog. 2. If block #412 fails checksum verification, the engine flags that specific block for surgical recovery from mirror parity, without touching the remaining 99.9% of healthy blocks. ```python import zlib # Compute Adler-32 or CRC32 per block: csum = zlib.adler32(block_bytes) ``` ## Pop Quiz Why do enterprise storage filesystems like ZFS and Btrfs maintain checksums for individual blocks rather than a single whole-file checksum? - [ ] Block checksums pinpoint the exact corrupted disk sector, enabling surgical repair of only the damaged block. - [ ] Whole-file checksums cannot be calculated on files larger than 100 megabytes. - [ ] Block checksums prevent magnetic heads from overheating during heavy read operations. - [ ] Because Linux hardware RAID controllers require 32-bit checksum tags on SATA cables. ## Real-World Challenge Implement: 1. `compute_block_checksums(data: bytes, block_size: int = 64) -> list[int]`: Computes `zlib.adler32` checksums across sequential chunks of `block_size`. 2. `scan_corrupted_blocks(data: bytes, expected_checksums: list[int], block_size: int = 64) -> list[int]`: Recomputes block checksums and returns a list of zero-indexed block numbers that fail verification. <details> <summary>Solution</summary> ```python import zlib def compute_block_checksums(data: bytes, block_size: int = 64) -> list[int]: checksums = [] total_len = len(data) for offset in range(0, total_len, block_size): chunk = data[offset : offset + block_size] checksums.append(zlib.adler32(chunk)) return checksums def scan_corrupted_blocks(data: bytes, expected_checksums: list[int], block_size: int = 64) -> list[int]: corrupted_indices = [] actual_checksums = compute_block_checksums(data, block_size) for idx, (expected, actual) in enumerate(zip(expected_checksums, actual_checksums)): if expected != actual: corrupted_indices.append(idx) # If length mismatch, record missing/truncated blocks if len(actual_checksums) < len(expected_checksums): for idx in range(len(actual_checksums), len(expected_checksums)): corrupted_indices.append(idx) return corrupted_indices # Verification original_file = b"ALPHA_BLOCK_0001" * 4 + b"BRAVO_BLOCK_0002" * 4 + b"CHARLIE_BLK_0003" * 4 assert len(original_file) == 192 # Exactly three 64-byte blocks baseline_checksums = compute_block_checksums(original_file, block_size=64) assert len(baseline_checksums) == 3 # Inject bit-flip corruption into block 1 (offset 75) corrupted_payload = bytearray(original_file) corrupted_payload[75] ^= 0x01 # Flip one bit in block 1 damaged_blocks = scan_corrupted_blocks(bytes(corrupted_payload), baseline_checksums, block_size=64) assert damaged_blocks == [1], f"Expected block 1 corrupted, found: {damaged_blocks}" ``` </details>