Chunked Stream Copier with sendfile Zero-Copy
Traditional file-to-socket transfers copy bytes from kernel disk buffers into Python userspace memory, and back down into kernel socket buffers. The os.sendfile syscall performs zero-copy transfers entirely inside the operating system kernel, eliminating CPU memory churn and context switches.
import os
# Zero-copy kernel space transfer:
# os.sendfile(out_fd, in_fd, offset, count)
# Chunked Stream Copier with sendfile Zero-Copy When Nginx or a high-performance Python media server streams a 4 GB video file to a user, naive code reads chunks into Python buffers and writes them out: ```python # The Slow Userspace Bounce: # Disk -> Kernel Page Cache -> Python Userspace Buffer -> Kernel Socket Buffer -> Network while chunk := f_in.read(65536): f_out.write(chunk) ``` This causes **four context switches** and **two unnecessary CPU memory copies** per chunk, burning CPU cycles and thrashing CPU cache lines. 🏎️⚡ The Linux `sendfile(2)` syscall provides **Zero-Copy**: data transfers directly from the kernel page cache into the destination descriptor inside kernel space. Python userspace never touches the bytes. ```python import os # Stream directly inside the kernel: bytes_transferred = os.sendfile(out_fd, in_fd, offset, count) ``` ## Pop Quiz Why does the `sendfile` syscall drastically reduce CPU utilization during high-volume static asset streaming? - [ ] Data transfers directly between kernel file buffers and network buffers without copying bytes into userspace application memory. - [ ] It compresses data blocks using hardware acceleration before transmission. - [ ] It disables TCP retransmission timeouts on gigabit local networks. - [ ] Because Python bypasses GIL locking by offloading bytecode directly to the network interface card. ## Real-World Challenge Implement `zero_copy_stream_copy(src_path: str, dst_path: str, chunk_size: int = 65536) -> int`. - Open both files with `os.open`. - Loop using `os.sendfile` until all bytes from the source file are transferred. - Return the total count of transferred bytes. Ensure descriptors close cleanly in `finally`. <details> <summary>Solution</summary> ```python import os def zero_copy_stream_copy(src_path: str, dst_path: str, chunk_size: int = 65536) -> int: src_size = os.path.getsize(src_path) in_fd = os.open(src_path, os.O_RDONLY) out_fd = os.open(dst_path, os.O_CREAT | os.O_WRONLY | os.O_TRUNC) total_transferred = 0 try: offset = 0 while total_transferred < src_size: count = min(chunk_size, src_size - total_transferred) copied = os.sendfile(out_fd, in_fd, offset, count) if copied == 0: break offset += copied total_transferred += copied return total_transferred finally: os.close(in_fd) os.close(out_fd) # Verification src_file = "/tmp/test_sendfile_source.bin" dst_file = "/tmp/test_sendfile_dest.bin" # Write 100 KB payload payload = b"HIGH_PERFORMANCE_ZERO_COPY_STREAM_" * 3000 with open(src_file, "wb") as f: f.write(payload) transferred = zero_copy_stream_copy(src_file, dst_file, chunk_size=16384) assert transferred == len(payload) with open(dst_file, "rb") as f: assert f.read() == payload os.remove(src_file) os.remove(dst_file) ``` </details>