Back
+20 XP
Concept

Embedded Database Part 3: Crash Recovery & Replay

When an embedded database process restarts after a crash or power cut, the in-memory index is lost. The engine recovers state by sequentially scanning the append-only log from byte zero, restoring the Keydir index 🛡️.

python
import io
import struct

def recover_keydir(log_bytes: bytes):
    stream = io.BytesIO(log_bytes)
    keydir = {}
    while True:
        offset = stream.tell()
        header = stream.read(8)
        if len(header) < 8:
            break  # Reached end of log file
        klen, vlen = struct.unpack("!II", header)
        key = stream.read(klen).decode()
        stream.seek(vlen, io.SEEK_CUR)  # Skip value
        keydir[key] = offset  # Latest write wins
    return keydir

print("Engine recovers complete in-memory index from disk log 🛡️")

# 112. Embedded Database Part 3: Crash Recovery & Replay When an embedded database process restarts after a crash or power cut, the in-memory index is lost. The engine recovers state by sequentially scanning the append-only log from byte zero, restoring the Keydir index 🛡️. ```python import io import struct def recover_keydir(log_bytes: bytes): stream = io.BytesIO(log_bytes) keydir = {} while True: offset = stream.tell() header = stream.read(8) if len(header) < 8: break # Reached end of log file klen, vlen = struct.unpack("!II", header) key = stream.read(klen).decode() stream.seek(vlen, io.SEEK_CUR) # Skip value keydir[key] = offset # Latest write wins return keydir print("Engine recovers complete in-memory index from disk log 🛡️") ``` Adding CRC32 checksums to each record header ensures corrupted entries created during abrupt power outages are detected instantly and excluded from the recovered index 🛡️! --- ### Question How does an append-only storage engine handle partially written trailing bytes caused by a sudden server crash? 🧠 - [ ] a. It detects corrupted or truncated bytes at the log tail and truncates the file back to the last valid checksum/offset 🛡️ - [ ] b. It terminates the operating system init system - [ ] c. It converts incomplete binary records into whitespace characters - [ ] d. It requests a full file backup from the local hardware RAID controller --- ### Challenge Build `recover(log_bytes)` returning `{key: offset}`. Test with log data for `[('x', '10'), ('y', '20'), ('x', '30')]`. Print `kd['x'] > kd['y']`. 🎯 <details> <summary>Solution</summary> ```python import io import struct def make_log(): s = io.BytesIO() for k, v in [('x', '10'), ('y', '20'), ('x', '30')]: kb, vb = k.encode(), v.encode() s.write(struct.pack('!II', len(kb), len(vb)) + kb + vb) return s.getvalue() def recover(log_bytes): s = io.BytesIO(log_bytes) kd = {} while True: off = s.tell() hdr = s.read(8) if len(hdr) < 8: break klen, vlen = struct.unpack('!II', hdr) key = s.read(klen).decode() s.seek(vlen, io.SEEK_CUR) kd[key] = off return kd kd = recover(make_log()) print(kd['x'] > kd['y']) ``` </details>

Checkpoint

How does an append-only storage engine handle partially written trailing bytes caused by a sudden server crash? 🧠

Practice

Build `recover(log_bytes)` returning `{key: offset}`. Test with log data for `[('x', '10'), ('y', '20'), ('x', '30')]`. Print `kd['x'] > kd['y']`. 🎯

Terminal
01
Tap code blocks below to assemble
Available Blocks