Back
+30 XP
Concept

Read-Only Archive (.zip) Indexer without Decompression

ZIP files store their table of contents at the end of the archive in the Central Directory. By seeking backwards to locate the End of Central Directory (EOCD) signature, an indexer reads file names, uncompressed sizes, and CRC-32 checksums instantly without decompressing any files.

python
import struct

# EOCD Signature: 0x06054b50
eocd_pos = data.rfind(b'\x50\x4b\x05\x06')
entries, cd_size, cd_offset = struct.unpack_from('<HHII', data, eocd_pos + 8)[:3]

# Read-Only Archive (.zip) Indexer without Decompression When querying a 10 GB zip archive containing 20,000 files in an AWS S3 bucket, downloading and unzipping the entire 10 GB just to check if `invoice_99.pdf` exists wastes time, money, and memory. 📦🔍 The **ZIP file format (APPNOTE.TXT)** places its catalog at the **end** of the file. By reading the last 64 KB of the file: 1. Locate the **End of Central Directory (EOCD)** record via signature `0x06054b50` (`PK\x05\x06`). 2. Read `total_entries`, `cd_size`, and `cd_offset`. 3. Jump to `cd_offset` and unpack the **Central Directory Headers** (`0x02014b50` = `PK\x01\x02`). You obtain the entire archive index—file names, compressed sizes, uncompressed sizes, and CRC checksums—in milliseconds without downloading or decompressing compressed data! ```python import struct EOCD_SIGNATURE = b"\x50\x4b\x05\x06" ``` ## Pop Quiz Why is the Central Directory placed at the end of a ZIP file rather than at the beginning? - [ ] Historical tape streamers could append files incrementally without needing to know total file sizes upfront. - [ ] To prevent operating systems from reading zip contents without third-party decryption drivers. - [ ] Because early microprocessors could only read binary files in reverse byte order. - [ ] To reserve the first 64KB of disk space for executable bootstrap loaders. ## Real-World Challenge Implement `index_zip_central_directory(zip_bytes: bytes) -> list[dict]`. - Scan backwards to find the EOCD signature `b"\x50\x4b\x05\x06"`. - Extract `total_entries`, `cd_offset`, and loop through the Central Directory file headers (`b"\x50\x4b\x01\x02"`). - Return a list of dicts: `[{"filename": str, "uncompressed_size": int, "crc": int}, ...]`. <details> <summary>Solution</summary> ```python import struct import io import zipfile def index_zip_central_directory(zip_bytes: bytes) -> list[dict]: eocd_sig = b"\x50\x4b\x05\x06" eocd_pos = zip_bytes.rfind(eocd_sig) if eocd_pos == -1: raise ValueError("Invalid ZIP file: End of Central Directory (EOCD) not found") # EOCD layout: 4B sig, 2B disk, 2B cd_disk, 2B entries_disk, 2B total_entries, 4B cd_size, 4B cd_offset disk, cd_disk, num_entries, total_entries, cd_size, cd_offset = struct.unpack( "<HHHHII", zip_bytes[eocd_pos + 4 : eocd_pos + 20] ) offset = cd_offset cd_sig = b"\x50\x4b\x01\x02" results = [] for _ in range(total_entries): if zip_bytes[offset : offset + 4] != cd_sig: break # Fixed central header is 46 bytes header_data = zip_bytes[offset : offset + 46] ( sig, ver_made, ver_need, flags, method, mtime, mdate, crc, comp_size, uncomp_size, fn_len, extra_len, comment_len, disk_start, int_attr, ext_attr, local_offset ) = struct.unpack("<IHHHHHHIIIHHHHHII", header_data) offset += 46 filename = zip_bytes[offset : offset + fn_len].decode("utf-8", errors="replace") offset += fn_len + extra_len + comment_len results.append({ "filename": filename, "uncompressed_size": uncomp_size, "crc": crc }) return results # Verification buffer = io.BytesIO() with zipfile.ZipFile(buffer, "w") as zf: zf.writestr("schema.sql", b"CREATE TABLE users (id INT);") zf.writestr("config.yaml", b"host: localhost\nport: 8080") raw_archive = buffer.getvalue() catalog = index_zip_central_directory(raw_archive) assert len(catalog) == 2 assert catalog[0]["filename"] == "schema.sql" assert catalog[0]["uncompressed_size"] == len(b"CREATE TABLE users (id INT);") assert catalog[1]["filename"] == "config.yaml" ``` </details>