Erasure coding promises twice the space of 3-way replication in Ceph. The full math says otherwise

In a Ceph cluster with 384 TB raw, 3-way replication leaves you 128 TB usable. Erasure coding 4+2 leaves you 256 TB, surviving the same two simultaneous failures. Twice the space on the same hardware: Ceph's own documentation says it that plainly. And yet we —who run Ceph in production, spread across several datacenters— still put VM disks on 3-way replication. It's not nostalgia. It's that space is only one of four calculations, and the other three almost never make it into the slide deck.

The math that makes it into the slides: space

First, what each one does. With 3-way replication (Ceph's default pool type), every object is written whole, three times, to three OSDs on three different hosts. You survive losing two copies because a third one always remains. The price: every TB of data consumes 3 TB of disk. A 3.0 factor; 33% usable space.

With erasure coding, the object is split into k data chunks plus m parity chunks. With a 4+2 profile, every object becomes 6 chunks on 6 hosts, and any 4 of them can rebuild the data: you survive losing 2. The overhead is (k+m)/k = 1.5; 67% usable space. It's RAID 6 promoted to a distributed system.

So far, the space math — and erasure coding wins it by a landslide: same iron, same tolerance, 128 more TB. If the analysis ends at this table, you migrate every pool this very afternoon. Let's continue with the math that doesn't get shown.

The write math

A write on 3-way replication is boring, and that's a compliment: the primary OSD receives the object and sends two full copies to its peers. Three writes, zero math, predictable latency.

A write on EC 4+2 splits the object, computes two parity chunks (CPU) and spreads six writes across six hosts. For large sequential writes, that cost amortizes well. The problem is the small random write —exactly what a VM or a database generates—: modifying 4 KB inside a stripe means reading chunks, recomputing parity and writing again. Where replication did three writes and slept soundly, EC stages a read-modify-write operation spread across the cluster.

Can you put a VM disk on an EC pool? Technically, yes: since Luminous there's allow_ec_overwrites (BlueStore only) to enable partial writes, and RBD works. But the question was never whether you can. It's whether you want to pay that toll on every small write of every VM, forever.

The recovery math

The day a disk dies is when you pay for the architecture. With replication, recovery is copying: the OSDs holding the surviving copies stream them over, done. With erasure coding, every lost chunk gets rebuilt: you read k chunks from other OSDs, redo the math and write the result. More reads, more east-west traffic and more CPU, at exactly the moment the cluster is degraded and your VMs keep demanding IOPS as if nothing happened. It's the difference between photocopying a page and rewriting it from four classmates' notes.

The node math (the one that decides it for an SMB)

This is the calculation that ends the conversation most often. With the failure domain at host level —the serious minimum—, an EC pool needs at least k+m hosts: 6 nodes for a 4+2. And the documentation itself recommends k+m+1 so the cluster can self-heal with one node down: 7 nodes. Replica 3 works with 3 and is comfortable with 4.

If your cluster has 3, 4 or 5 nodes —most of the virtualization clusters we see at SMBs—, the decision is made before you start: there aren't enough failure domains for serious EC. Small profiles like 2+1? They exist, and they're the poor man's RAID 5: a single tolerated failure and a rebuild with the whole cluster holding its breath. We don't put m=1 in production with data that matters, and Ceph's documentation doesn't encourage adventures either: for block and file it suggests staying at m=3 or below, and in general not choosing k>4 or m>2 unless you fully understand the consequences.

Where erasure coding does shine

None of this makes erasure coding bad technology. It's an excellent tool for the right job: data written once, in large blocks, and mostly read.

What we do

In short

Replica 3 and erasure coding don't compete: they do different jobs. Replication buys low latency and simple recovery, paying in disk; erasure coding buys space, paying in CPU, network, complexity and nodes. The trap is looking only at the space column, which is the only one that shows up in the budget calculator. One pool for VMs, another for cold data, and all four calculations done before creating either: that's storage design, and there's no shortcut.

We've been operating Ceph distributed storage under production Proxmox clusters for years, and we've made this decision —which pool, which profile, how many nodes— many times with real data on the table. It's the kind of decision that's cheap to think through and very expensive to undo.

Sources (verified): (k+m)/k overhead, the "free lunch" quote, the 4+2 profile offering "double the usable space compared to replication with size=3", the m≤3 recommendation and the advice not to exceed k=4/m=2 without understanding the ramifications, allow_ec_overwrites on BlueStore, and the k+m failure-domain requirement (k+m+1 recommended) — official Ceph documentation; EC pool creation and the replicated metadata pool on Proxmox VE (pveceph pool create --erasure-coding, EC lacking full OMAP support for RBD) — Proxmox VE documentation (pveceph). The 384 TB example is arithmetic on those formulas.