Three servers wired to each other, two NICs each, and not a single 25 GbE switch on the invoice. Euro for euro it is one of the best builds you can put together today. The Proxmox wiki has documented it for years and offers four ways to wire it. The failure-scenarios section holds the detail that decides which one you pick: with two of the four, the day a cable breaks the nodes end up seeing different things.
The wiki's exact words: "the nodes now have a different view, Node 2 can communicate with Node 3, while that is not possible for Node 1 anymore. You will see behavior such as Ceph showing the services on Node 3 to be down". What counts is in the second half. Nothing on Node 3 has broken. Its disks are spinning, its OSDs are up and its data is intact. It is just that the one place you happen to be looking from is the one that can no longer see it.
That is the difference between a failure and an outage, and here you buy it when you choose the wiring, months before anybody trips over the cable.
Everybody rehearses losing a node
The rehearsal that does get done is powering off a server. And it goes fine: the wiki itself says so without drama —if Node 3 goes down, "the Ceph and Proxmox VE cluster will remain functioning, though with reduced redundancy". It is the comfortable scenario, because the loss is total and symmetrical: the node is gone, and everyone still standing agrees that it is gone.
The uncomfortable scenario is a different one: the loose cable, the SFP that runs hot, the port that negotiates at 1 Gb instead of 25, or the technician who uses a site visit to "tidy things up a bit". There the loss is partial and asymmetric, and that is far worse than a total loss, because each node builds a coherent answer out of incomplete information and the three answers disagree. The screen does not say "a cable failed": it shows a service down in a place where nothing is down.
The four ways of wiring three nodes
The build is always the same triangle: each server with at least two NICs, each plugged straight into one of the other two. What changes is what you run on top of those cables. The wiki describes four options, and what really decides between them is not performance: it is what happens when a link goes down, not a node.
- Routed (simple). Each packet goes only to the addressed node. Needs no extra software and gives the best performance of the four. If a link goes down there is no alternative path: split view.
- Routed (with fallback). The same, with a routing protocol on top: if the direct link drops, traffic goes via the middle node. Performance "similar" to simple; one link down means "a bit higher latency" and "the cluster will stay fully functional". It holds.
- RSTP. A loop with rapid spanning tree. It requires Open vSwitch, because the Linux bridge only carries STP "without the rapid", and the difference is not cosmetic: in Proxmox's own tests RSTP recovers "within a few seconds" while STP takes around 30 seconds, which is "long enough for Ceph to start to complain and throw some warnings". It holds, with a permanent toll: while the loop is complete, RSTP keeps an artificial cut-off between two nodes, so traffic between those two always goes via the third.
- Broadcast. Every packet is sent to both other nodes. Easiest to set up, heaviest on bandwidth. If a link goes down: split view, same as routed simple.
Our preference, stated plainly and without pretending it is the only defensible one: routed with fallback. Not because it is the most elegant, but because it is the only one of the four whose failure mode is "it got slower". A system that degrades in speed can be explained at three in the morning; one that degrades in coherence burns half an hour of diagnosis before somebody remembers to look at a cable. RSTP lands somewhere similar, but it drags Open vSwitch into the equation, and every piece you add to the data path is one more to maintain.
And one line from the wiki worth reading before falling in love with the savings: "We recommend to use switches for clusters larger than 3 nodes or if a 3 node cluster should be expanded in the future". The mesh is an answer for three nodes. If the two-year plan is to reach five, the switch belongs in this year's budget, not next year's.
What changed: now the hypervisor does the routing
Until recently, building the routed-with-fallback setup meant installing FRR by hand, setting fabricd=yes in /etc/frr/daemons, writing one frr.conf per node and inventing a unique IS-IS NET for each of them. It works perfectly well and plenty of people run it that way. But it lives outside the cluster: not in the web interface, not in the access control, and not replicated.
Proxmox VE 9.0, on 5 August 2025, brought it inside under the name fabrics: routed networks between nodes, configurable from Datacenter → SDN → Fabrics, with OpenFabric and OSPF. 9.1 (19 November 2025) put them in the resource tree reporting routes, neighbours and interfaces. And 9.2, on 21 May 2026, added two more protocols and the fine-grained control that was missing:
- BGP as a fabric, in eBGP unnumbered mode: each node with its own autonomous system number, peering with its neighbours over the physical interfaces "without requiring IP addresses on the fabric links".
- WireGuard as a fabric, for encrypted tunnels between nodes or with third parties. With a caveat the documentation does not hide: "It does not provide dynamic routing by itself" —it is transport, and the dynamic routing comes from OSPF or BGP on top.
- Prefix lists and route maps as first-class SDN objects with their own API, plus route redistribution on OSPF fabrics and the ability to dry-run a change before applying it.
Four fabric protocols inside the hypervisor panel in nine and a half months: three that route and one that only transports. For the case at hand the wiki recommends the simplest one: "Proxmox VE currently supports OSPF and OpenFabric. For simple setups, such as this one, we recommend OpenFabric, as it's easier to set up than OSPF". That page was last edited on 27 March 2026, two months before BGP and WireGuard arrived; a good reminder that when two official pages disagree, the one with the later date wins.
The procedure fits in a paragraph: create the fabric, give it a name —eight characters maximum on OpenFabric, a detail you discover late— and an IPv4 prefix; add each node with its router-ID inside that prefix and tick the two mesh interfaces. If those interfaces have no IP they run unnumbered and inherit the node's IP. Then Apply Configuration, which generates and applies the FRR configuration on each machine, and check:
vtysh
show openfabric neighbor
show openfabric routeThen, when you install Ceph, the fabric shows up in the drop-down of available networks in the initial configuration step. One knob is worth turning knowingly: the Hello and CSNP intervals, which the wiki example lowers to 1 and 2. The text states the price without dressing it up —"Lower numbers mean that the network will fail over faster in the case of link failures, but nodes will generate more traffic"— and it is worth deciding with that sentence in front of you.
The label that comes in the box
This sits on the first page of the SDN chapter of the official documentation, under "Current Status". Three lines, three different things:
Core SDN, which includes VNet management and its integration
with the Proxmox VE stack, is fully supported.
IPAM, including DHCP management for virtual guests, is in tech preview.
Complex routing via FRRouting and controller integration are in tech preview.The third line is the one that concerns us, with a precision that has to come first: Proxmox does not write the word "fabrics" there, it writes "complex routing via FRRouting", which is exactly what a fabric is underneath. The equivalence is ours. What is not ours is the second mention: the project's published roadmap sets as a goal "Bring the remaining tech-preview parts of the SDN stack to full production readiness".
Our reading, and it is not the only possible one: that is not a reason to avoid it. FRR is FRR, it has been moving real networks for years, and what is in preview is the Proxmox integration layer on top. But it is a reason to decide with your eyes open what you hang off it, and above all not to learn about the label on the day something behaves oddly. There is an enormous difference between "we are taking this risk because it pays off" and "we did not know there was a risk". The first sentence is said in a meeting; the second, in a postmortem.
And corosync, does that go here too?
It is the obvious temptation: you already have 25 GbE between the three nodes, so why not put the cluster traffic on it too? The Proxmox documentation answers this unambiguously:
ideally corosync runs on its own physically separated network.
Especially do not use a shared network for corosync and storage
(except as a potential low-priority fallback in a redundant configuration).The mesh is the storage network. And the figures attached to that recommendation are specific: the cluster stack "requires a reliable network with latencies under 5 milliseconds", and above around 10 ms with more than three nodes stability "gets rather unlikely". Corosync does not need bandwidth, it needs nobody injecting jitter; a 25 Gb Ceph recovery doing backfill is exactly the kind of neighbour that does. We wrote in > about why that clock is not a fixed number but grows with every node you add.
Translated into purchasing: the two fast NICs for the mesh are not all the NICs you need. You also need at least one dedicated NIC for the primary corosync link —1 Gb is plenty— plus the management port. Anyone sizing the server by counting only the 25 Gb ports ends up sharing exactly what the documentation asks you not to share. As for how a three-node Ceph behaves when it loses a piece, we covered that separately in >.
The four files that change hands
This is the part that appears in no screenshot and that, in companies where "network" and "systems" are different people —or different providers—, decides whether the fabric is an improvement or an argument. Enabling it moves four files from one side of the table to the other:
/etc/pve/sdn/— the source of truth, and it belongs to the cluster. Prefix lists, route maps and fabrics live here, replicated. They are edited from the interface or the API, go through a pending state and do nothing until somebody hits apply./etc/network/interfaces— written by the SDN. That is why adding or updating a node inside a fabric also requiresSys.Modifyon that node: the documentation justifies it by saying the operation "involves writing to the node's /etc/network/interfaces file". With a sharp edge: if you remove an interface whose entry was set tomanual, the IP "will not get removed on applying the SDN configuration"./etc/frr/frr.conf— generated. The cluster configuration is translated into FRR stanzas and ends up here, on every node. If you came from the manual build, the wiki warns about the collision: "This will not work in combination with the EVPN functionality from the Proxmox VE SDN as it will overwrite our FRR configuration"./etc/frr/frr.conf.local— the seam. This is where your own per-node additions go, appended on top of what was generated. It is the only point in the set where what Proxmox writes and what you write coexist, and therefore the first place to look when something does not add up.
There are also two things the fabric creates without asking that are worth recognising when they turn up in an ip route. A dummy interface with the router-ID as its address, acting as a loopback and passive from birth. And, per fabric, an automatically generated access list and route map that rewrite the source address of outgoing packets: thanks to that, a ping between nodes leaves from the loopback rather than from the physical interface. Several names are reserved and cannot be reused for your own —only_default, loopbacks_ips, correct_src and anything starting with pve_.
The question this map opens is an organisational one: who signs off a change to those four files? Because from the moment the fabric exists, part of the routing table is touched from the virtualisation panel, with virtualisation permissions, by the people who administer virtualisation. If that is not said out loud beforehand, the first person to find out is whoever is on call.
Three warnings buried in the small print
All three are written in the official documentation, all three are easy to skip, and all three cost you an afternoon.
- IPv6 only works on OpenFabric, and it requires global IPv6 forwarding. There is no convenient per-interface switch as there is for IPv4, so the documentation's recipe is a
post-up sysctl -w net.ipv6.conf.all.forwarding=1. The text itself warns that this affects how interfaces handle SLAAC, neighbour advertisements and router advertisements. One fabric setting that changes the IPv6 behaviour of the whole machine. - Prefix lists and route maps deny by default at the end. That is standard FRR semantics, but it surprises anyone coming from a different tool. The documentation even shows the pattern: a trailing unconditional
permitentry, because "without it the implicit deny at the end of the route map would drop all unmatched routes". - If the nodes are virtualised —lab, proof of concept, training— each node's fabric interfaces must sit on separate bridges or different VLANs. Otherwise, says the wiki, "routing loops may occur". Which means: the lab where you were going to rehearse this is precisely where it is easiest to build it wrong.
What we are not saying
- ✗That the mesh replaces a switch in every case. It replaces the switch for three nodes that are going to stay three. The recommendation to use switches above three nodes, or if there is a plan to grow, is Proxmox's, not ours, and we think it is right.
- ✗That "tech preview" means broken. It means the vendor does not yet commit to that layer at the same level as the rest, and says so in its own roadmap. Reading the label is not the same as rejecting the feature: it is deciding with the fact on the table.
- ✗That we have timed the four variants on your hardware. The numbers above are the ones Proxmox publishes, quoted as they stand. What we add is the judgement: which failure mode we would accept and why. Anyone selling you a failover time for your cluster without having measured it in your cluster is quoting you theirs.
If you already have the mesh and do not know which of the four it is, the answer is one look away: if there is an frr.conf with openfabric in it, or a fabric in the SDN panel, you have fallback. If there is a bond in broadcast mode or a few static routes in /etc/network/interfaces, you are in the group that splits on one cable. Neither of the two is an emergency today. But one of the two is worth knowing before somebody touches a patch lead.
Sources (checked 12 Oct 2026): the four mesh variants, their behaviour when a link is lost, the RSTP versus STP timings, the Hello/CSNP intervals and the OpenFabric recommendation come from Full Mesh Network for Ceph Server (official Proxmox wiki, last edited 27 Mar 2026). The "tech preview" status of complex routing, the fabrics, the permissions, prefix lists and route maps, the unnumbered mode, the dummy interface, the reserved names and the IPv6 note come from the SDN chapter of the Proxmox VE documentation. The corosync quotes and the 5 and 10 ms thresholds come from the cluster chapter. Release dates, the 9.0, 9.1 and 9.2 changes and the goal about the preview parts come from the official roadmap. Every quoted sentence is verbatim from those pages. We are not Proxmox or VMware resellers and we sell licences for neither: we run Proxmox VE with Ceph in production and we write from there.
Do you know what happens in your cluster if a cable comes loose tonight?
You do not find that out during an outage: you decide it at design time. We design and run infrastructure and cloud on Proxmox VE, we build the distributed Ceph storage underneath it and, as a network operator running our own network, we do the networks and communications part ourselves instead of leaving it between two providers.
Talk to everyWAN
Microsoft 365
ES 
CAT 
EN