← Back to Blog
DistributedSystems Backend gfs distributed-systems hadoop hdfs system-design file-systems

The Google File System: How Google Stores Petabytes on Cheap, Unreliable Machines

How the Google File System stores petabytes on cheap, unreliable machines: chunking, replication and heartbeats across thousands of disks.


Contents

    Every operating system you’ve used — Windows, Linux, macOS — manages files the same basic way: it divides the disk into fixed-size blocks (often 4 KB), and every file you save is split across as many blocks as it needs.

    number of blocks required = total file size / block size
    

    A 12 KB photo on a 4 KB block size file system needs 3 blocks. Those 3 blocks don’t have to be next to each other on disk — they can be scattered anywhere. So the file system keeps a main index (like a table of contents) that records which blocks belong to which file, and in what order, so it can reassemble the file when you open it.

    This works great on a single machine. But what happens when your “file” is actually the entire internet’s worth of web pages, and your disk isn’t big enough — no matter how large a single machine you buy?


    The Problem: One Disk Isn’t Enough

    Google’s early problem was simple to state and brutal to solve: they needed to store and process enormous amounts of data (web crawls, logs, indexes) that no single machine’s disk could hold.

    The obvious fix — “just add a bigger, more expensive server” — doesn’t scale:

    • It’s costlier: enterprise-grade, high-capacity storage hardware is expensive.
    • It doesn’t fix scalability: eventually you hit the biggest disk money can buy, and you still need more.
    • It creates a single point of failure: if that one expensive machine dies, everything on it is gone.

    So instead of one giant reliable machine, Google took the opposite approach: use thousands of cheap, commodity machines, expect them to fail regularly, and design the software to tolerate that. That software is the Google File System (GFS), published by Ghemawat, Gobioff, and Leung in 2003.


    The Core Idea: Split Files Into Chunks

    GFS applies the same idea as a local file system, just at a massive scale. Instead of 4 KB blocks, a file is split into chunks of 64 MB each. Each chunk is stored as a plain file on a regular machine called a chunkserver.

    Sitting in front of all the chunkservers is a single master server. When a client wants to read or write a file, it doesn’t fetch the data through the master — it just asks the master “which chunkservers hold the chunks for this file?”, and then talks directly to those chunkservers to move the actual data. This keeps the master out of the way of heavy data traffic.

    graph TD
        Client[Client]
        Master["Master Server<br/>(namespace + chunk mapping table)"]
        CS1[Chunkserver 1]
        CS2[Chunkserver 2]
        CS3[Chunkserver 3]
    
        Client -- "1. Which chunkservers have chunk X?" --> Master
        Master -- "2. Chunkserver locations" --> Client
        Client -- "3. Read/write chunk data directly" --> CS1
        Client -- "3. Read/write chunk data directly" --> CS2
    

    Since there are now thousands of chunkservers instead of one disk, a failure is no longer rare — it’s expected. Handling that reality is the entire point of GFS’s design.


    Replication: Surviving Machine Failures

    If each chunk only existed on one chunkserver, losing that machine would mean losing that piece of data forever. GFS solves this with replication: every chunk is copied to multiple chunkservers, typically 3 (the “replication factor” or RF).

    graph LR
        Chunk["Chunk A"]
        Chunk --> S1["Chunkserver 1<br/>(replica)"]
        Chunk --> S2["Chunkserver 2<br/>(replica)"]
        Chunk --> S3["Chunkserver 3<br/>(replica)"]
    

    [!NOTE] Replication isn’t just about surviving crashes — it also spreads read traffic across multiple machines, so popular chunks don’t bottleneck on a single server.

    But replicas can go down too. GFS needs a way to notice when that happens and fix it automatically.

    Heartbeats Keep the Master’s Map Up to Date

    Every chunkserver periodically sends a heartbeat message to the master. The heartbeat reports which chunks that server currently holds and confirms it’s still alive. The master uses these heartbeats to keep its in-memory chunk mapping table (which chunkserver holds which chunk) accurate at all times.

    If a chunkserver misses its heartbeats, the master assumes it’s dead. It then checks: does this chunk still have enough healthy replicas to match the replication factor? If not, the master picks another chunkserver and copies the chunk there — automatically re-replicating until the RF is satisfied again.

    sequenceDiagram
        participant CS as Chunkserver
        participant M as Master
        participant CS2 as New Chunkserver
        CS->>M: Heartbeat (I'm alive, here are my chunks)
        M->>M: Update chunk mapping table
        Note over M: If a chunkserver stops sending heartbeats...
        M->>M: Replica count falls below RF, pick a new chunkserver
        M->>CS2: Instruct: copy chunk here
    

    This self-healing loop is what makes GFS resilient — no human has to notice a dead disk and manually replace the data; the system keeps chasing the target replication factor on its own.


    What If the Master Itself Dies?

    The master is a single machine holding critical metadata: the file namespace, the file-to-chunk mapping, and (via heartbeats) the chunk locations. If it’s a single point of failure, the whole design falls apart.

    GFS doesn’t keep a live “hot backup” that can instantly take over writes. Instead, it protects the master’s metadata in two ways:

    • Operation log replication: every metadata change (creating a file, adding a chunk, etc.) is written to an operation log, which is flushed and replicated onto several remote machines before the change is considered committed. The master also periodically writes a checkpoint so recovery doesn’t require replaying the entire log.
    • External monitoring restarts the master: if the primary master crashes, monitoring infrastructure outside GFS itself detects the failure and starts a brand-new master process on one of the machines holding the replicated log. Clients find the master through a canonical DNS name, so once that name points at the new process, failover is transparent to them.
    flowchart LR
        Primary["Primary Master"] --> Remote1["Replicated log copy 1"]
        Primary --> Remote2["Replicated log copy 2"]
        Monitor["External Monitoring"] --> NewMaster["New Master Process"]
        Remote1 --> NewMaster
        Remote2 --> NewMaster
    

    GFS also runs read-only shadow masters. They apply the replicated operation log a little behind the primary, so they can keep serving metadata reads (like directory listings or slightly stale file lookups) even while the primary is down or being restarted — they just can’t serve writes.


    Why 64 MB Chunks Instead of 4 KB Blocks?

    This is the detail that surprises most people coming from regular file systems. GFS deliberately uses a much larger chunk size than a typical OS block:

    • Fewer chunks per file means the master’s metadata table stays small enough to fit in memory, even for huge files.
    • Fewer round trips to the master: a client that wants to read a large file only needs to ask the master for chunk locations once in a while, not for every few KB.
    • GFS was built for large, sequential reads/writes (think: multi-gigabyte log files and web crawl data), not for lots of small random-access files — so big chunks match the workload.

    Quick Comparison: Local File System vs. GFS

    Aspect Local File System Google File System
    Unit of storage Block (e.g., 4 KB) Chunk (64 MB)
    Stored on One disk Thousands of chunkservers
    Failure handling None (single disk) Replication (RF, typically 3)
    Metadata owner OS file index Single master (operation log replicated to multiple machines)
    Failure detection N/A Heartbeats between chunkservers and master
    Master failure N/A New master process replays the replicated log; shadow masters serve reads meanwhile

    The Legacy: HDFS

    GFS was never open-sourced, but its design directly inspired the Hadoop Distributed File System (HDFS), which follows the same blueprint: a NameNode (playing the role of GFS’s master) tracks metadata, while DataNodes (playing the role of chunkservers) store replicated blocks of data. If you’ve worked with Hadoop, Spark, or any big-data pipeline, you’ve already been relying on ideas straight out of the GFS paper.


    The Bottom Line

    GFS proved that you don’t need expensive, reliable hardware to build a reliable storage system — you need software that assumes failure is normal and designs around it:

    1. Split large files into big chunks (64 MB) instead of tiny blocks, to keep metadata small and reads efficient.
    2. Replicate every chunk across multiple chunkservers so no single failure loses data.
    3. Use heartbeats to continuously detect dead servers and automatically re-replicate chunks to maintain the target replication factor.
    4. Replicate the master’s operation log to multiple machines, so metadata — the map to everything — survives even if the master process goes down, with read-only shadow masters riding out the outage.

    The next time you hear “we store petabytes of data across thousands of commodity servers,” this heartbeat-driven, self-healing chunk architecture is very likely doing the heavy lifting behind the scenes.


    References

    1. The Google File System — Ghemawat, Gobioff, Leung (SOSP 2003)
    2. Apache Hadoop Distributed File System (HDFS) Architecture