NVMe standing in for RAM: memory tiering works if half of yours is idle

In the write-up of the latest VMware Explore there is a sentence that explains the moment better than any chart: DRAM now often costs several times more than the servers it lives in. VMware's answer to that is called memory tiering: if part of a host's RAM has gone hours without anybody touching it, let it drop to a local NVMe drive and free up room for what is actually in use. It ships, it is documented, and two sessions on it filled 500-seat rooms. The least quoted part of all this is a recommendation VMware itself published on its Cloud Foundation blog on 6 August: keep active memory below 50% of physical memory. Read backwards, that recommendation is a diagnosis.

What it actually does

Broadcom's definition is one line long: "Memory Tiering allows you to add memory capacity to an ESX host by using NVMe devices that are installed locally on the ESX host as tiered memory". The hypervisor watches which memory pages are cold —nobody has read or written them for a while— and demotes them to NVMe. When somebody asks for them again, they come back up. To the virtual machine all of this is invisible: it still sees its 64 GB.

The feature shipped as available with VMware Cloud Foundation 9.0, and this year, in 9.1, VMware tuned it to behave better with databases and added dashboards showing how much tiering activity there is, plus management tooling. The vSphere 9.0 documentation asks for vCenter and ESX 9.0 or later, and the feature ships deactivated. And there is a stated roadmap: Dave Morera, staff technical marketing architect at VMware, put it this way at Explore: "we have a two-to-three year roadmap". What is on that roadmap says a fair amount about what is missing today, and we come back to it below.

Tiering is not having more memory

This is our own judgement and it is worth saying early, because the headline invites the wrong reading. Memory tiering does not create memory: it trades capacity for latency. A DRAM access and an NVMe access are not on the same scale; there are orders of magnitude between them, and no algorithm fixes that distance. What the algorithm does is bet that you will almost never pay that toll, because what you ask for is almost always upstairs.

The bet is reasonable. In any cluster with a few dozen machines there is memory doing nothing: staging environments powered on since March, application servers sized "just in case", the VM from a project that got cancelled and nobody switched off. That memory is allocated, it takes up space, and nobody touches it. It is exactly the material tiering moves to NVMe without anyone noticing. The bet breaks when the machine that ended up downstairs is the one that starts working all at once at nine in the morning.

The number that decides is yours, not VMware's

In an article from 6 August about memory reservations, VMware drops the most useful operational recommendation in all the material we have read on this feature: keep active memory below 50% of total physical memory. It is a best practice published by the vendor, and here is our reading of it: that sentence describes which kind of host this is for. If your active memory sits above that threshold, tiering will not save you; it will move your problem over to latency.

The problem is that almost nobody knows their own figure. When we walk into someone else's infrastructure and ask how much memory is genuinely in use, the usual answer is the one on the summary tab, allocated memory, which measures something else. Measuring it is not hard and costs nothing: take several weeks of active-memory history, month-end closes included, and compare it against installed physical memory. That ratio decides whether this feature is a good purchase or an expensive patch. We argued the same thing when we wrote about the rise in RAM prices: before signing off on modules, find out how much of what you already have is working.

The requirements are a shopping list

Broadcom's documentation and the Explore write-up leave a list of conditions worth reading in one go, because taken separately each one looks minor:

• The NVMe has to be local. Verbatim: "NVMe devices cannot be over fabric or Ethernet. They must be installed locally". So much for solving it from the array you already own.
• And not just any NVMe will do. Broadcom asks for the vSAN cache spec, mixed use, 3 DWPD. At Explore it got more specific still: a minimum of 100,000 writes per second and 7,300 TB of lifetime writes.
• Turning it on or off requires the host in maintenance mode. Meaning: drain the host and move everything. This is not a switch you try out on a Tuesday afternoon.
• Some things go away. With memory tiering enabled, the documentation says Quick Boot is not supported on that host and neither is suspending VMs to memory; hot-plug and "prepare to remove" are out for the NVMe devices doing the tiering, not for everything else. Suspend-to-disk and snapshots keep working. The host also cannot carry Intel Optane or NVDIMM-N persistent memory.
• There is a ceiling. By default the system limits tiered capacity to the size of DRAM, a 1:1 ratio, with a maximum of 4 TB per partition. The default and the recommended starting point is that 1:1; VMware allows raising it to 1:4, and when it does it points at specific cases such as virtual desktops, not as a general setting.

Lined up, the list stops looking like a feature and starts looking like a budget: high-endurance enterprise NVMe drives in every host, a maintenance window per server, and the subscription up to date. The requirements have a reason, because that NVMe is going to be written to a lot, and often. But it is the part that does not fit in the headline. The answer to hardware getting expensive involves buying more hardware, of a kind that has not escaped the rise either: the same market forecasts we cited in our round-up of memory prices put NAND at +70-75% quarter on quarter in Q2 2026, off the same demand surge as DRAM.

The 25% almost nothing is known about

The figure VMware gave at Explore is that memory tiering currently supports around 75% of workloads. Credit for putting a number on it. The catch is that nobody publishes what the remaining 25% is. The only part of that group with a name of its own sits on the roadmap: monster VMs —machines of up to 960 vCPUs and 16 TB of memory spread across several physical hosts— are slated for VCF 9.2, around May 2027.

And here we should be precise, even if it makes the argument less tidy: a 16 TB machine spread across several hosts is an extreme case, and a small company's ERP or the reporting engine that spikes at month-end almost certainly fall inside the 75% that is already supported. We are not going to imply otherwise to make the criticism land better. What we do maintain, and this is our own judgement, is that an aggregate percentage does not answer the only question that matters to you: whether the specific machine squeezing your memory is inside or outside that 75%. That list is not published, so the answer comes from testing on a host, not from reading a datasheet.

And no, a reservation does not get you out

The reflex of anyone with years on vSphere is: "fine, I'll give the database a full reservation and stop worrying". It does not work like that, and VMware says so in its own 6 August article: "it does not pin memory to DRAM". A reservation stops the host reclaiming that memory through swapping, ballooning or compression, but "those reserved pages can still be satisfied from either DRAM or NVMe". The distinction they draw is a good one: the reservation number is a promise to the scheduler; pinned memory is a contract with the hardware. They are not the same thing.

Genuinely taking a VM out of tiering means disabling it for that VM, and that has an effect the same article illustrates with an example: disable it and reserve the VM fully, and the ratio for the rest of the host shifts from 1:1 to 1:2. Translated: the DRAM you guaranteed the important machine, you took from its neighbours, who now lean twice as hard on NVMe. It is a perfectly legitimate decision —some workloads deserve it— but it is a redistribution, not a gift. Anyone who thinks they can armour three or four critical VMs and leave the rest untouched will find the cost in the place they were not looking.

When it does pay off

With all of the above on the table, the feature still solves a real problem. In our view it pays off when three things line up at once: your active memory sits clearly below half of physical; the cluster has lots of medium, lukewarm VMs rather than two monsters; and you are already inside VCF on the right version, so the incremental cost is drives and maintenance windows rather than a new platform. In that scenario, putting NVMe in each host is considerably cheaper than putting in DRAM, and it fits the advice we have been giving for months: stretch the hardware you already have rather than buying at the worst moment in the market.

The two figures that will get repeated most in presentations are 30% fewer CPU cycles and 40% lower total cost of ownership. Those are vendor claims, reported in the Explore write-up, and we have not measured them. We are neither calling them false nor repeating them as if they were ours: if anyone is going to use them to justify a purchase, ask for the detail of which workload and against which baseline they were obtained, because a TCO percentage without a starting scenario means nothing.

What we do before touching anything

When the "we are out of memory" conversation arrives, the first thing we look at is the gap between allocated memory and touched memory, with history. Almost always the same things show up: powered-on VMs nobody claims, environments that outlived their project, sizings copied from a template. Powering things off and right-sizing does not make for a nice slide, but it is free and it does not add a new component that can fail. Only after that does it make sense to talk about ballooning, page deduplication —with its downsides, and it has them: it costs CPU— or, now, tiering to NVMe.

And one lever is still unnamed, the one we have most often seen actually move the bill: licensing rather than engineering. We have already written that the mass exodus from VMware never happened and that much of the market renegotiated rather than left, and also that there are cases where we do NOT recommend migrating to Proxmox. We are nobody's reseller: not VMware's, not Proxmox's, and we sell licences for neither. What we do say, just as calmly, is that if you are looking at this feature because hardware got expensive, your licence calendar deserves to sit in the same spreadsheet as the drives.

At everyWAN we design and run virtualized infrastructure on both platforms and have for years, we do VMware to Proxmox migrations when the numbers work, and we say no when they do not. If what you need is somebody to look at your numbers before you sign anything, that is consulting, and it is billed by the hour, not by the licence.

Sources (verified 2 September 2026): the VMware Explore write-up, the roadmap, the 75%-of-workloads figure, the definition and timing of monster VMs (up to 960 vCPUs and 16 TB spread across several physical hosts, slated for VCF 9.2 around May 2027), the NVMe requirements (100,000 writes per second and 7,300 TB of lifetime writes), the 30%-fewer-CPU-cycles and 40%-lower-TCO figures, the Dave Morera quote and the 500-seat rooms all come from The Register, 1 September 2026. The definition of the feature, the default 1:1 ratio with a 4 TB maximum, the local-NVMe requirement ("cannot be over fabric or Ethernet"), the vSAN cache spec with 3 DWPD, maintenance mode, the incompatibility with Intel Optane and NVDIMM-N, and the limitations on Quick Boot, suspend-to-memory and hot-plug are in Broadcom's vSphere 9.1 documentation. The recommendation to keep active memory below 50% of physical —which is not in the product documentation but on the corporate blog, where it appears tied to the effect of full reservations on the DRAM budget—, the sentence "it does not pin memory to DRAM", the distinction between a promise to the scheduler and a contract with the hardware, and the 1:1-to-1:2 shift example come from "Memory Tiering and VM Memory Reservation", 6 August 2026, on the official VMware Cloud Foundation blog. Initial availability with VCF 9.0 is announced on the same blog. Our own judgement, not reported fact: that tiering trades capacity for latency and does not create memory; reading the 50% figure as a diagnosis of which host the feature suits; that the requirements list works as a shopping list; that the machines with a memory problem tend to be the big ones, precisely those on the roadmap rather than in today's product; and the order of levers we apply (measure allocated against touched, power off and right-size, and only then tier). The up-to-1:4 ratio and its fit with cases such as virtual desktops come from the VCF 9.0 availability announcement and the Explore write-up, not from the documentation page cited. The NAND figure (+70-75% quarter on quarter in Q2 2026) is a TrendForce forecast already gathered and verified in our post on memory prices, linked in the body. The performance and TCO figures are vendor claims and we have not verified them with our own measurements.