Back
+30 XP
Concept

Fast Directory Inode Scanner with os.scandir

Scanning directories with os.listdir followed by os.path.isfile triggers an expensive stat(2) syscall for every file entry. Using os.scandir returns DirEntry iterators that yield cached inode file attributes directly from the directory stream, achieving up to 20x speedup.

python
import os

with os.scandir(path) as it:
    for entry in it:
        # Cached without extra stat syscalls:
        if entry.is_file():
            print(entry.name, entry.stat().st_size)

# Fast Directory Inode Scanner with os.scandir If you run `os.listdir("/var/data")` on a directory with 100,000 files and then check `if os.path.isfile(f): size = os.path.getsize(f)`, your script executes **200,000 individual system calls** to the kernel. On network filesystems (NFS) or high-latency cloud storage, this takes minutes. 🚀📁 Python 3.5 introduced `os.scandir()`, which hooks into the underlying POSIX `getdents64` system call. The kernel returns directory entry names along with their file types (`d_type`) and inode stats in a single bulk operation. `entry.is_file()`, `entry.is_dir()`, and `entry.stat()` reuse the cached kernel data without issuing additional syscalls, delivering up to **20x faster** scans. ```python import os with os.scandir(folder) as entries: for entry in entries: if entry.is_file(follow_symlinks=False): size = entry.stat(follow_symlinks=False).st_size ``` ## Pop Quiz Why is `os.scandir()` dramatically faster than `os.listdir()` when scanning directories with 50,000 files? - [ ] It extracts file types and inode data directly from the kernel directory stream, avoiding 50,000 separate stat syscalls. - [ ] It indexes files by reading the Master File Table directly into GPU memory. - [ ] It creates asynchronous threads for every directory level automatically. - [ ] Because os.listdir() encrypts filenames before returning them to Python. ## Real-World Challenge Implement `fast_tree_summary(root_dir: str) -> dict`. - Use `os.scandir` to traverse directories recursively without invoking `os.listdir` or `os.path.join`. - Return `{"file_count": int, "dir_count": int, "total_bytes": int}`. <details> <summary>Solution</summary> ```python import os def fast_tree_summary(root_dir: str) -> dict: file_count = 0 dir_count = 0 total_bytes = 0 stack = [root_dir] while stack: current_dir = stack.pop() try: with os.scandir(current_dir) as iterator: for entry in iterator: try: if entry.is_dir(follow_symlinks=False): dir_count += 1 stack.append(entry.path) elif entry.is_file(follow_symlinks=False): file_count += 1 total_bytes += entry.stat(follow_symlinks=False).st_size except OSError: pass except OSError: pass return { "file_count": file_count, "dir_count": dir_count, "total_bytes": total_bytes } # Verification summary = fast_tree_summary("/tmp") assert isinstance(summary["file_count"], int) assert isinstance(summary["dir_count"], int) assert isinstance(summary["total_bytes"], int) assert summary["file_count"] >= 0 ``` </details>