Upgrading a Proxmox VE 8.4 Cluster on LINSTOR/DRBD to Proxmox VE 9: The Lab, the Night, and Everything We Learned

The short version#

If you only read one section, make it this one.

• Build a lab that runs your exact package versions and your exact configs, then rehearse the whole thing more than once. Our first full rehearsal turned up twelve problems. The dress rehearsals found more. Production found only small ones.
• Verify every DRBD disk before you start. drbdadm verify found 488 blocks on 11 disks where the replicas really differed, and none of them were false alarms. Find that out before you change a single package.
• Consider a full outage instead of a rolling upgrade. In the lab, DRBD 9.3.1 lost writes on a replica when a resync overlapped a running guest. With every VM shut down, that risk disappears, and so does a lot of work.
• Upgrade DRBD to 9.3.4 or later before anything else. 9.3.3 and 9.3.4 fix out-of-sync bits that never clear, and replicas that can end up "UpToDate" with different data.
• Kernel 7.0 has no crct10dif. If LINSTOR auto-picked it as your DRBD verify-alg (ours did), expect drbd@linstor_db to fail on the first 7.0 boot. It recovers by itself, but it's alarming if you don't expect it.
• Never upgrade the LINSTOR packages on the node running the controller. Evict it first with drbd-reactorctl evict linstor_db.
• Name your apt hold file *.pref (or give it no extension). Otherwise apt silently ignores it.
• Make sure the DRBD module built for the new kernel before you reboot, not after.
• If you run a Full Mesh with FRR, the classic post-up ... restart frr.service line can deadlock boot on PVE 9, and ifreload strips your loopback address. Plan for both.
• HA has opinions during a full shutdown. We lost two parallel stop requests, and HA fenced a node whose resource manager had just been restarted by the upgrade. Neither hurt anything, but you'll want to know why.

The cluster we upgraded#

The cluster is a fairly typical hyperconverged setup:

• Hardware: three Dell servers, each with two Xeons and about 500 GiB of RAM. They boot from Dell BOSS cards using legacy BIOS and GRUB.
• Storage: LINSTOR on top of DRBD 9, on local NVMe. One storage pool keeps two copies plus a diskless tiebreaker; the other keeps three copies.
• Controller: the LINSTOR controller runs as an HA service managed by DRBD Reactor. It keeps its database on a small DRBD resource called linstor_db and fails over between nodes.
• Storage network: a 100 Gb full mesh between the three nodes, with no switch, using FRR and OpenFabric. This is the Proxmox wiki's "Full Mesh" setup, built by hand.
• Uplinks: 10 Gb to two top-of-rack switches.
• Guests: about 50 running VMs on roughly 19 TiB of replicated disks, 17 of them under Proxmox HA.

Here's what changed:

Why this upgrade is harder than a stock one#

A stock Proxmox upgrade is mostly apt dist-upgrade and a reboot. LINSTOR and DRBD add four complications:

1. DRBD is a kernel module built outside the kernel tree. A new kernel means building DRBD for it with DKMS, and that build has to work before you reboot. Kernel 7.0 is newer than anything LINBIT's changelog explicitly lists as supported.
2. LINSTOR doesn't do rolling upgrades. A satellite on a different version from the controller shows up as OFFLINE(VERSION MISMATCH), even for a patch release. So the order in which you touch nodes matters.
3. Your storage network is software. If the mesh drops for a few seconds, DRBD loses quorum, and depending on your settings the guests get I/O errors. The upgrade touches FRR, ifupdown2 and systemd, which are exactly the pieces that keep the mesh up.
4. Proxmox's tooling doesn't know about any of it. pve8to9 will flag the DKMS module and not much else.

None of these is a showstopper. But each one turns "follow the wiki" into "rehearse it until it's boring," which is what we did.

Step one: build a lab that lies to you as little as possible#

We built the rehearsal lab as three nested Proxmox VMs running on the production cluster itself. Each lab node had 4 vCPUs, 8 GB of RAM, four virtual NICs and a few virtual disks, stored on production's own LINSTOR storage.

The goal was to make the lab boringly identical to production wherever we could:

• Same packages, to the exact version. Every Proxmox, kernel, FRR and LINBIT package matched. Of 831 packages, 763 were identical; the rest were Debian point releases the archive no longer offered.
• Same configs. Production's /etc/network/interfaces byte for byte except the addresses, the same FRR config, the same corosync link priorities, and the same LINSTOR storage pools, resource groups, properties and HA controller setup.
• The same warts, on purpose. Production had a few config problems the runbook was supposed to fix: an ip_forward setting in /etc/sysctl.conf (PVE 9 ignores that file), systemd-boot installed on a GRUB system, no LINSTOR client config, and an FRR hostname left at its default. We copied every one of them into the lab, so the fixes got rehearsed along with everything else.

We also wrote down what the lab could not tell us:

• anything about the real hardware (firmware, the BOSS boot card, NIC naming on kernel 7.0);
• real timings (a lab node reboots in about a minute, a production node in five and a half);
• one networking difference. The 100 Gb mesh ports became point-to-point VXLAN tunnels in the lab, because OpenFabric forces every link to point-to-point and can't share one virtual segment.

Canaries: measure what the guests feel#

"The upgrade worked" isn't the same as "the guests never noticed." So every lab node ran a small Debian VM, a canary, with a writer that does this:

• writes 4 KiB with O_DIRECT|O_DSYNC every 100 ms, to a file on a three-copy disk and to a raw two-copy disk;
• reports every write error, every write that takes longer than a second, and a read-only root file system;
• sends all of it to the serial console, where a recorder on the host stamps each line with the host's time. A paused guest's clock drifts, so its own timestamps can't be trusted.

Later runs added canaries in harder spots: one outside HA with onboot set, and one on a node that holds no copy of its own disk. The canaries are how we discovered the data-loss problem described further down. Without them, we'd have shipped the original rolling-upgrade plan.

Snapshots and resets#

Every lab node had a snapshot called baseline-84 (fresh PVE 8.4, configured like production), and a rollback took about two minutes. That made it cheap to run the whole upgrade, throw it away, fix the plan and run it again.

One lesson came straight out of building this: don't shut down an entire Proxmox HA cluster while an HA guest is still running. With shutdown_policy=migrate, the last node has nowhere to send the guest, and it gets fenced. Our snapshot script now sets HA resources to ignored, shuts the guests down, and waits until every node's resource manager is idle before stopping anything.

Rehearse with production's scripts, not lab copies#

By the last runs, the lab ran production's own scripts, with only the site-specific values (addresses, node names) swapped in from a config file. We had one rule: no fixes on the fly. Any deviation during a rehearsal counted as a bug in the plan. We fixed it in the plan and scripts, then ran the whole thing again.

Here's how the runs went:

Two warnings about putting a lab on the production cluster:

• It shares production's network and storage. Lab traffic crosses production's switches, and every lab write is replicated. Production's nightly backup once caused a finding in the lab.
• The reverse happens too. A lab reset recreated the lab VMs' disks on production LINSTOR in the middle of a production drbdadm verify, and one verify worker failed with "No such resource". Don't reset the lab while anything runs on production.

One more: run long jobs in tmux or as systemd units on a node, never from a laptop. During one rehearsal, the laptop driving it went to sleep for 23 minutes.

Rolling upgrade or full outage? The lab made the call#

The original plan was a classic rolling upgrade: drain a node with HA live migration, upgrade it, bring it back, move on. It's what everyone does, and it means no downtime.

The third rehearsal ended that. With a canary writing on every host, we found two things.

Replicas missed writes during resyncs. Of 20 checksum comparisons, three found blocks where one replica had the guest's data and another had zeros, on DRBD 9.3.1 and on 9.3.4. When a resync starts, DRBD 9.3 can briefly stop replicating to the node being resynced (the log shows replication( yes -> no )). Writes that land in that window reach the other copies, and the target is supposed to catch up from the bitmap afterwards. Every case we saw had a guest writing while two diskful secondaries were involved: a three-copy disk, or a two-copy disk whose VM ran on the tiebreaker node. With the guest paused, it never happened.

Guests got I/O errors in drained windows. While one node is down for its upgrade, every disk has one replica fewer. A network blip of one to two seconds was enough to lose quorum, and with on-no-quorum io-error the guest's root file system started failing writes until it was rebooted.

On top of that, about 30% of the rehearsal's time went into keeping guests running: drains, migrations, checks between drains. That time is pure overhead in an outage.

So we switched to a full outage in one night, with every VM shut down. With no guest writing, no resync can race a write, and a drained window can't cost anyone a file system. The cost was about three hours of planned downtime, plus every VM cold-booting into a new QEMU version. That was an easy trade for this client.

A few related decisions:

• One night, not two. We rehearsed both. Two nights gives you a soak on the new DRBD before the big jump, but costs an extra shutdown and start-up. One night was about 35 minutes shorter in total outage.
• Two reboots per node anyway. The first reboot lands the latest 8.4 with DRBD 9.3.4; the second lands PVE 9. Between them there's a checkpoint: if anything looks wrong there, you're on a fully working, fully patched 8.4 cluster and can stop for the night.
• Node order: least important first, the busiest node last. We started with the node that held neither the LINSTOR controller nor the HA manager's master role, and finished with the one that held both.
• Hard cut-off times. "Checkpoint passed by T+1:45. No PVE 9 upgrade starts after T+2:45. Start-up begins by T+3:15, no matter what." Deciding these in advance takes the heroics out of a 2 a.m. decision.

Before the night: know your DRBD#

Check every node's view, not just your own#

DRBD tracks out-of-sync data per connection, and each node only sees its own connections. A check that runs drbdadm status on one node can say "all clean" while two other nodes disagree about a disk. Check every node.

This loop prints every peer connection, on every node, that has out-of-sync data:

for n in pve1 pve2 pve3; do
  ssh root@$n "drbdsetup status --json" | jq -r --arg n "$n" '
    .[] | .name as $res | .connections[] | .name as $peer | .peer_devices[]
    | select(."out-of-sync" > 0)
    | "\($n) -> \($peer)  \($res)/\(.volume)  \(."out-of-sync") KiB out of sync"'
done

Our script also flags any local disk that isn't UpToDate (or Diskless), any connection that isn't Connected, any replication state that isn't Established, and any suspended I/O. It prints RESULT: CLEAN only when there's nothing at all. We ran it before every node, after every reboot, and before starting the guests.

Verify every disk before you start#

Out-of-sync bits only tell you what DRBD knows is different. drbdadm verify reads both replicas and compares checksums, so it can find what DRBD doesn't know about. We verified all 19 TiB the day before the upgrade, and found 488 4 KiB blocks on 11 disks where the replicas really differed.

We traced them back to some maintenance reboots a week earlier, still on DRBD 9.3.1. During those reboots, each node's copies were resync targets for a few seconds while VMs kept writing on other nodes. That's the same mechanism the lab had found. Two VMs that had no local copy had been reading a mix of both replicas.

The next section covers how to run the verify and how we repaired the blocks. The point here: do this before the upgrade, while you still have time to think. We repaired them in the first few minutes of the outage, with every VM off.

The rest of the prep list#

• A fresh backup the same evening (why backup alone isn't a recovery plan). We ran the nightly backup job by hand a couple of hours early and disabled the scheduled run for the night.
• Pre-download the packages (apt-get --download-only full-upgrade), so a slow mirror can't eat your window.
• Stage firmware. Our 100 Gb cards needed a firmware update. We burned it online with mstflint before the first reboot, so that reboot activated it at no extra cost. We backed up the old image first, as the rollback.
• Out-of-band access. Have the iDRAC/IPMI console open, and the Proxmox ISO ready as a rescue boot.
• Clear onboot on the guests, so no node reboot can start a VM halfway through the night. Record which guests had it, and set it back at the end.

The DRBD and LINSTOR gotchas#

These are the problems we hit, in roughly the order they'll bite you.

1. Out-of-sync bits that never clear (DRBD 9.3.1)#

Symptom: a disk shows UpToDate on every node, connections are Established, no resync is running, and yet a few MiB stay marked out of sync forever. It happened after unclean reboots and after clean ones. Once, the kernel log showed DRBD trying to finish a resync while the bitmap was locked; exactly the resynced amount stayed set.

What makes it dangerous:

• Online verify gives false positives while those bits are set. We saw 433 blocks, then 0, then 0, then 757 on the same disk, while an md5 of every MiB of all three replicas (with the guest paused) was identical.
• The bits can also be real. Sometimes they mark writes a replica genuinely missed.
• Reconnecting behaves differently by version. On 9.3.1, a disconnect/reconnect simply drops the bits: "cleared bm UUID and bitmap". If the replicas really differ, you've just thrown away the record of where. On 9.3.4 the same reconnect reconciles them and resyncs.

Fix: go to DRBD 9.3.4 or later before anything else. The DRBD ChangeLog for 9.3.3 and 9.3.4 describes these cases:

• peers holding the same current UUID but carrying standing out-of-sync bits;
• two nodes ending up UpToDate with different data after such a reconnect;
• a divergence around the end of a resync.

In a mixed pair, the 9.3.4 side kept its bits through two reconnects and resynced them as soon as its peer ran 9.3.4. That's the behaviour you want.

Rule for 9.3.1: never "clear" standing bits by reconnecting until you've proven, with checksums, that the replicas are identical.

2. Kernel 7.0 has no crct10dif#

Symptom: on the first boot into kernel 7.0, drbd@linstor_db.service fails with VERIFYAlgNotAvail ("failed to allocate crct10dif for verify"), and the satellite's first adjust fails too.

Cause: we never set a verify-alg. LINSTOR picked one automatically from its allowed list, and crct10dif was first, so every resource used it. Kernel 7.0 doesn't have it.

What happens next: about 15 seconds later the satellite regenerates its resource files with an algorithm that exists, and everything connects. Nodes still on 6.8 keep crct10dif. DRBD connects across that difference, but online verify refuses to run between a 6.8 node and a 7.0 node, because both ends need the same algorithm. Our LINSTOR 1.35 ended up choosing sha512 everywhere.

What to do:

• Clear the failed unit with systemctl reset-failed drbd@linstor_db.service.
• After the upgrade, check that every resource uses one algorithm, and not crct10dif:

grep -h -o -E 'verify-alg[[:space:]]+"?[a-z0-9-]+' /var/lib/linstor.d/*.res | tr -d '"' | sort | uniq -c

3. Never upgrade LINSTOR on the node running the controller#

Symptom: when you upgrade the LINSTOR packages on the node where the controller is active, the package's prerm script stops linstor-controller. DRBD Reactor reacts immediately. In the lab we saw both outcomes:

• On one node, Reactor failed the controller over to a node still on the old version. The freshly upgraded node then showed OFFLINE(VERSION MISMATCH).
• On another node, Reactor restarted the controller in place while dpkg was swapping the jar files. The controller died with a FileNotFoundException for the old jar, and the CLI and the Proxmox plugin got "connection refused" until a manual systemctl restart linstor-controller.

DRBD and the guests didn't notice either way, but you lose the control plane at exactly the wrong moment.

Fix: upgrade the nodes that don't run the controller first. Then move the controller off the last node and upgrade it:

drbd-reactorctl status linstor_db     # find the node where it's active
drbd-reactorctl evict linstor_db      # run on that node; another node takes over
linstor controller version            # confirm the controller answers from its new home

Keep in mind that LINSTOR doesn't support rolling upgrades. Even a 1.33.3 satellite against a 1.33.1 controller shows VERSION_MISMATCH, so do all three nodes back to back.

We did LINSTOR in two hops:

• 1.33.1 → 1.33.3 on PVE 8.4, before anything else;
• 1.33.3 → 1.35.3 only after all three nodes ran PVE 9.

In between, a hold kept apt from jumping ahead, which brings us to the next problem.

4. Your apt hold file might be silently ignored#

Our first hold file was called linstor-hold-1.33. apt only reads files in /etc/apt/preferences.d/ that have no extension or the .pref extension. To apt, that name has the extension .33, so it ignored the file without a word and offered LINSTOR 1.35.3 in the middle of the PVE 9 upgrade.

The fix is simply the name, /etc/apt/preferences.d/linstor-hold-1.33.pref:

Package: linstor-controller linstor-satellite linstor-common
Pin: version 1.33.3-1
Pin-Priority: 1001

Always check that a hold actually holds. Before every dist-upgrade:

apt-cache policy linstor-satellite          # Candidate must be the version you pinned
apt-get -s dist-upgrade | grep -E '^Remv'   # nothing important may be removed

We stopped the upgrade if the candidate was wrong, or if the simulation removed any of proxmox-ve, pve-manager, linstor-*, drbd-* or frr.

5. Check the DKMS build before you reboot#

If drbd-dkms fails to build for the new kernel, you find out after the reboot, when the node comes up without its storage. Check before the reboot instead:

new=$(find /lib/modules -mindepth 1 -maxdepth 1 -printf '%f\n' | sort -V | tail -n 1)   # the newest kernel
dkms status drbd                     # must list drbd 9.3.4 as "installed" for $new
modinfo -k "$new" -F version drbd    # must print 9.3.4

If it doesn't print the version you expect, don't reboot. Check that the kernel headers for the new kernel are installed (LINBIT recommends installing the headers before drbd-dkms). Then rebuild with apt install --reinstall drbd-dkms. If it still won't build, pin the old kernel with proxmox-boot-tool kernel pin and stay on it until it does.

DRBD 9.3.4 built and ran fine on 7.0 for us, in the lab and in production. But no LINBIT changelog states 7.0 support, so treat it as something to check, not something to assume.

6. The empty peer name#

During the assessment we found something odd: 56 of 85 DRBD resource files on one node said on "" for one of its peers. drbdadm status showed node-id:N instead of the peer's name, and drbd-reactorctl status said the controller was "active on node ''".

The real damage showed up at boot. DRBD Reactor ran adjust with the bad file about 20 seconds before the satellite regenerated it, and drbd@linstor_db failed.

• Fix: restart linstor-satellite, which regenerates the files. Do it on a node that isn't currently running the controller.
• Check: grep -l 'on ""' /var/lib/linstor.d/*.res is empty, and drbdadm --dry-run adjust all prints nothing.

We never found the root cause. The files dated from right after a cluster-wide boot months earlier.

7. DRBD shuts down before HA has moved the guests#

Symptom: a reboot without draining the node first stalled guest writes on other nodes for about 10 seconds, and left out-of-sync bits behind.

Cause: at shutdown, systemd stopped drbd-graceful-shutdown.service (drbdadm down all) before pve-ha-lrm had migrated the HA guests away. With guests still running, drbdadm down fails with "Device is held open by someone", and it's never retried. So the peers just wait for DRBD's ping timeout.

Fix: a drop-in that makes DRBD shut down after the guests are gone. Systemd stops units in reverse start order, so it says Before=:

# /etc/systemd/system/drbd-graceful-shutdown.service.d/zz-after-pve-guests.conf
[Unit]
Before=pve-ha-lrm.service pve-guests.service

Run systemctl daemon-reload, and check with systemd-analyze verify that you didn't create an ordering cycle. After this, DRBD tore down cleanly every time.

8. on-no-quorum: io-error or suspend-io?#

LINSTOR clusters usually run DRBD with quorum majority. The question is what a node does when it loses quorum. We measured both settings in the lab with a 12-second network isolation and a writer every 100 ms:

So suspend-io looks better, but it has a nasty failure mode. Suppose a node loses only the storage mesh: corosync stays up over the management network, so Proxmox HA sees nothing wrong. That node's VMs freeze until someone runs drbdadm secondary --force or reboots it. After somewhere between 30 seconds and three minutes, most guests start timing out anyway.

Our decision:

• Keep io-error normally.
• Switch to suspend-io only for attended maintenance windows, at the controller level, and switch back afterwards:

linstor controller set-property DrbdOptions/Resource/on-no-quorum suspend-io
# ... the maintenance ...
linstor controller set-property DrbdOptions/Resource/on-no-quorum io-error

• Keep linstor_db on io-error always. DRBD Reactor needs the controller to fail cleanly so it can move it.

Also note: DRBD's own default is suspend-io, and drbdsetup show hides default values. Use drbdsetup show --show-defaults when you check what's really in effect.

9. A network reload can take down your storage#

This one's really a networking problem, but it shows up as a storage outage, so it belongs here too.

On a Full Mesh setup, ifreload -a (which the GUI's "Apply Configuration" and every SDN apply run) removes the /32 address that FRR put on the loopback. The wiki's post-up ... restart frr.service line restarts FRR to bring it back. That restart cut our node off the mesh for about 8 seconds, longer than DRBD's network timeout. DRBD dropped both peers, quorum was lost for 11–15 seconds, and with io-error the guests got I/O errors.

It happened on production once, during network work a week before the upgrade: a network reload on one node made linstor_db lose quorum, and the controller failed over. And because frr.service allows only three starts in three minutes, a fourth apply in quick succession leaves FRR stopped.

The cure is in the networking section below. Until then: only change networking on a node with no running guests.

Finding and fixing out-of-sync disks#

This is the part we'd most want to read before our own upgrade, so here it is in detail.

Running drbdadm verify at scale#

Online verify reads a resource on both ends of one connection, checksums the blocks, and marks every block that differs as out of sync. It repairs nothing. That's good: you decide what's right.

Here's how we ran it. (The examples use resource names like pm-1a2b3c4d, which is how the linstor-proxmox plugin names its resources.)

Pick one verifying node per disk. For a two-copy disk, verify from either copy-holding node against the other. For a three-copy disk, verify one node against both peers. We sorted the disks largest first and spread them over the nodes by load.

Raise the resync rate for the verify, then put it back. Verify is limited by c-max-rate (about 100 MiB/s per connection by default). We set it per resource through LINSTOR, so the satellite doesn't overwrite it. drbdadm verify returns immediately while the kernel does the work, so wait until no peer device shows a Verify state before you put the rate back:

res=pm-1a2b3c4d
linstor resource-definition drbd-options --c-max-rate 512000k "$res"
drbdadm verify "$res:pve2"                 # once per peer
while drbdsetup status "$res" | grep -qE 'replication:Verify[ST]'; do sleep 5; done
linstor resource-definition drbd-options --unset-c-max-rate "$res"

Run several at once. A single verify ran at about 90–135 MiB/s no matter how high we set the cap. So we split each node's share into three workers running in parallel, which put most streams at 130–290 MB/s.

• On our NVMe, three workers per node barely touched CPU or iowait.
• The full 19 TiB took about three and a half hours.
• One 1.8 TiB disk took 2.7 hours on its own.

Run the workers as systemd units, not from your SSH session. Then a dropped connection or a sleeping laptop can't kill them. Here, verify-worker.sh stands for a script that runs the loop above for each disk it's given. For a scheduled run, systemd-run --on-calendar gives you a one-shot timer:

systemd-run --unit verify-a --on-calendar '2026-11-07 06:00:00' \
  bash -c '/root/verify-worker.sh pm-1a2b3c4d pm-5e6f7a8b >>/root/verify-a.log 2>&1'

Reading the results. DRBD writes the results to the kernel log, not to the command's output:

• every run ends with Online verify done (total N sec; ...), or ... done but N 4k blocks skipped when the guest was writing to blocks during the check;
• only when it finds something do you also get Online verify found N 4k blocks out of sync, plus one Out of sync: start=S, size=C line per range, in 512-byte sectors.

Parse all three, or you'll treat "found nothing" and "didn't finish" the same way.

Treat verify as a pointer, not proof.

• Skipped blocks (one to six per disk, on busy VMs) weren't checked at all. Cover them with a second pass, or verify with the VM off.
• Standing out-of-sync bits on 9.3.1 make verify unreliable (see gotcha 1). Verify only when every connection shows 0 out of sync.
• Before you repair anything, prove the difference by comparing the actual bytes.

Proving a difference#

To compare copies, each node has to read its own copy. Reading the DRBD device doesn't guarantee that, because DRBD may serve the read from a peer. So read the backing device, the LVM volume under DRBD (drbdsetup show <res> lists it as disk). LINSTOR puts DRBD's metadata at the end of the backing device, so sector N of the DRBD device is sector N of the backing device. Read only up to the DRBD device's size, and the metadata stays out of the comparison.

For a small disk, the bluntest method works. Pause or stop the guest, then on every node with a copy, hash the whole disk one MiB at a time:

res=pm-1a2b3c4d
dev=/dev/drbd1005                              # the DRBD device: used for its size
bd=/dev/linstor_vg/pm-1a2b3c4d_00000           # its backing device on this node
size=$(blockdev --getsize64 "$dev")
dd if="$bd" bs=1M count=$((size / 1048576)) iflag=direct status=none \
  | split -b 1M --filter=md5sum > "/root/md5-$res-$(hostname).txt"

Line N of the output is MiB N of the disk. Copy the files to one place and diff them. With the guest paused, a 4 GiB disk took about 15 seconds.

For a multi-TiB disk, hash only the ranges verify reported (dd with skip and count in 4 KiB blocks, sector ÷ 8), on each node's backing device, and compare. Then look at the bytes themselves. In our case one side had real data, and the other had zeros or older data.

Deciding which copy is right#

This is a judgment call, so make it explicitly:

• If the VM runs on a node with a local copy, that copy is what the guest has been reading and writing. Use it as the source.
• If the VM runs diskless (on a tiebreaker or a node without a copy), it may have been reading a mix of copies. Look at the differing regions and pick the copy that's newer by content, or the one the out-of-sync bits say has the data.
• If you can't tell, stop and look harder. Our lab tooling refused to choose when no node held bits, or when the bits pointed both ways.

Three ways to repair#

Option 1: reconnect, when the copies are already identical. If checksums prove the replicas match and all that's left is stale out-of-sync bits, a reconnect clears them:

drbdadm disconnect pm-1a2b3c4d:pve2
drbdadm connect pm-1a2b3c4d:pve2

Only do this when both disks are UpToDate and replication is Established. And never on 9.3.1 without the checksum proof, because there a reconnect throws the bits away (gotcha 1).

Option 2: a full resync, when the copies differ and the disk is small. With the VM shut down, tell DRBD the bad copy is invalid, and it copies the whole disk from a good one:

drbdadm invalidate pm-1a2b3c4d                 # on the node with the bad copy
# or, from the node with the good copy:
drbdadm invalidate-remote pm-1a2b3c4d:pve2

That's simple and thorough, but it copies the entire disk. A 1 GiB disk resynced in about 12 seconds in the lab; a 2 TiB disk takes hours.

Option 3: a block-level rewrite, when the copies differ in a few places on a big disk. This is what we used in production. The trick: when you write a block on a DRBD Primary, DRBD replicates that write to every peer. So if you read the good copy's blocks and write them straight back to the same place through the DRBD device, every replica ends up with the good data. Only the affected blocks move.

With the VM shut down, on the node with the good copy:

res=pm-1a2b3c4d
dev=/dev/drbd1005
drbdadm primary "$res"
# For each "Out of sync: start=S, size=C" range (512-byte sectors, 4 KiB aligned):
dd if="$dev" of="$dev" bs=4096 skip=$((S / 8)) seek=$((S / 8)) count=$((C / 8)) \
   iflag=direct oflag=direct conv=notrunc status=none
sync
drbdadm secondary "$res"

A few details make this safe:

• Make sure the reads come from the local copy. Check the resource's read-balancing (we required prefer-local), so DRBD doesn't serve reads from a peer.
• Use 4 KiB-aligned ranges, and direct I/O on both sides.
• No Primary anywhere else, which means the VM is off, and every peer Connected.
• Re-hash every range on every copy afterwards. On DRBD 9.3.1, the rewrite fixed the data but left verify's out-of-sync marks in place. Once the hashes match everywhere, a disconnect/reconnect of each marked pair (option 1) clears those marks. After that, our checker said CLEAN, and a second verify found 0.

In production, the whole repair, 488 blocks on 11 disks across three nodes, took about two minutes at the start of the outage. We had planned it, rehearsed it and checked it the afternoon before.

What not to do#

• Don't resync while the guest writes. That's exactly the window where the lab lost writes.
• Don't repair until you know which copy the guest has been reading.
• Don't trust verify while out-of-sync bits are standing, and don't trust a single verify on a busy VM.

The runbook, phase by phase#

Here's the shape of the night, generalized. Every step on every node ended with a gate: a script that printed GO only if all of these held:

• cluster quorum, all nodes online;
• HA services stable (in a full outage, stopped counts as stable);
• drbd-check-all CLEAN on every node;
• no failed systemd units we didn't expect;
• the LINSTOR satellites all Online.

Phase 0: before the night#

• Verify every DRBD disk and plan the repairs.
• Take a fresh backup and disable the scheduled job.
• Pre-download packages and stage firmware.
• Stage every script and config file on the nodes, and diff them against your source the same afternoon.
• Open the out-of-band consoles.

Phase 1: cluster-wide housekeeping on 8.4#

1. Shut down the guests, clear onboot, and repair any out-of-sync disks while everything is off.
2. Fix the small host items:
   • move ip_forward from /etc/sysctl.conf to /etc/sysctl.d/, since PVE 9 ignores the old file;
   • remove systemd-boot on GRUB systems;
   • install the DRBD shutdown-order drop-in (gotcha 7);
   • add a linstor-client.conf that lists all three controllers.
3. Upgrade LINSTOR to the latest patch of your current minor version, passive nodes first, then evict the controller and do its node (gotcha 3).
4. Test corosync failover: block each link in turn and confirm the cluster keeps quorum on the other.

Phase 2: the latest 8.4, per node#

1. Gate.
2. Evict the LINSTOR controller if it's on this node.
3. Stage anything that needs to land in the new initramfs (we renamed NICs; see the networking section).
4. Burn firmware if needed.
5. apt full-upgrade. This brings DRBD 9.3.4, newer drbd-utils, DRBD Reactor and the LINSTOR Proxmox plugin.
6. Check the DKMS build for the new kernel (gotcha 5).
7. Reboot, then gate again before the next node.

The checkpoint#

On all nodes, before going any further:

• cat /sys/module/drbd/version shows 9.3.4;
• pve8to9 --full shows 0 FAIL (we had two expected WARNs: the DKMS module and intel-microcode);
• drbd-check-all is CLEAN.

If anything here isn't right, stop. You're on a healthy, patched 8.4 cluster, and the guests can come back up on it.

Phase 3: PVE 9, per node#

The official guide covers the Proxmox side. Here's what we added for LINSTOR/DRBD and for our rollback.

1. Evict the LINSTOR controller if it's active on this node.

2. Take a rollback point. On an LVM-thin install, snapshot the root LV. GRUB's boot code lives outside it, so copy that too. Also tar up /etc and /var/lib/linstor.d, and copy the tarball off the node:

dd if=/dev/sda of=/root/mbr-boot-code.bin bs=440 count=1   # plus the BIOS boot partition, if you have one
lvcreate -s -L 15G -n root-pre-pve9 pve/root
tar czf /root/pre-pve9-etc.tgz /etc /var/lib/linstor.d

The snapshot was nearly half full right after the dist-upgrade, so size it generously and watch lvs until you drop it.

3. Switch the repositories to trixie, in deb822 format.

• Remove the old sources.list entries, pve-enterprise.list and ceph.list.
• Add debian.sources, proxmox.sources and (client-only) ceph.sources for trixie. For Debian, include non-free-firmware.
• PVE 9 ships pve-enterprise.sources enabled. Without a subscription, apt update returns a 401, so set Enabled: false in it.

4. Point LINBIT at its PVE 9 suite, and keep the LINSTOR hold in place:

deb [signed-by=/etc/apt/trusted.gpg.d/linbit-keyring.gpg] https://packages.linbit.com/public/ proxmox-9 drbd-9

5. Simulate first. Run apt-get -s dist-upgrade, and stop if it removes anything in proxmox-ve, pve-manager, linstor-*, drbd-* or frr, or if the LINSTOR candidate isn't the held version.

6. apt dist-upgrade. In the lab, this upgraded about 680 packages per node and took about ten minutes. Expect these prompts and choices:

• keep your lvm.conf. It contains the filter that stops LVM from scanning DRBD devices (r|^/dev/drbd|), and LINBIT also recommends rejecting /dev/mapper/[lL]instor;
• keep /etc/issue and /etc/default/grub;
• review sshd_config;
• take the maintainer's chrony.conf.

Also install intel-microcode (or amd64-microcode). pve8to9 FAILs on PVE 9 without it.

7. Check the DKMS build for the new kernel again (gotcha 5).

8. Reboot. Expect the crct10dif failure on the first 7.0 boot (gotcha 2); it heals itself in about 15 seconds.

9. Run the post-reboot checks:

• the DRBD module matches the running kernel;
• the satellite and DRBD Reactor are active, and all satellites are Online;
• no resource is Inconsistent, Outdated, StandAlone or stuck Connecting;
• every LINSTOR resource for the node exists in drbdsetup status (the satellite needs about 20 seconds after boot).

10. After the first node, stop for a go/no-go. It's your hardware canary. Did the boot card behave on the new kernel? Any machine-check errors in dmesg? Is DRBD up and the node gating GO? Ours passed 26 minutes ahead of its cut-off.

Phase 4: after all nodes run PVE 9#

1. LINSTOR to the new minor version:

   • remove the hold;
   • upgrade the passive nodes, then evict the controller and upgrade its node;
   • then upgrade the linstor-proxmox plugin.

   Ours took about two minutes.

2. Check the verify algorithm: one algorithm everywhere, and not crct10dif. Run systemctl reset-failed for any units left failed from the first 7.0 boot.

3. Check HA. On PVE 9, HA groups become HA rules. The migration runs automatically about two minutes after the last node joins, and nofailback becomes failback 0. Don't edit HA groups during the upgrade.

4. Move the mesh to SDN Fabrics if you're doing that (see the networking section).

5. Run the optional extras: the LVM autoactivation migration script (it changed nothing for us), and apt modernize-sources for any .list files left over.

6. Start the guests in your planned order. Set ha-rebalance-on-start to 0 during the start-up, or HA will shuffle guests while you're starting them; restore it afterwards. Then restore onboot, re-enable the backup job, and run a final drbd-check-all.

Networking: FRR, the full mesh and PVE 9#

If your DRBD runs over a routed full mesh, this section matters as much as the storage one.

The post-up FRR restart can deadlock boot on PVE 9#

The Proxmox Full Mesh wiki has you add a post-up line that restarts frr.service after the interfaces come up. On PVE 9, that can deadlock the boot. Guard it so it only restarts FRR if FRR is already running:

post-up /usr/bin/systemctl is-active --quiet frr.service && /usr/bin/systemctl restart frr.service || true

But first, run systemctl enable frr. On our nodes FRR was disabled, and that old post-up line was the only thing starting it at boot. With the guard in place and FRR disabled, a node came up with its mesh down.

ifreload strips your loopback address#

As described in gotcha 9, ifreload -a purges the /32 that FRR puts on lo, and restarting FRR to fix it costs you DRBD quorum. The tested fix on a hand-built mesh is to tell ifupdown2 to leave the loopback's addresses alone:

auto lo
iface lo inet loopback
    address-purge no

Do this together with removing the post-up restart. In the lab, removing the restart without address-purge no took the whole mesh down.

SDN Fabrics, and the timers PVE doesn't set#

PVE 9's SDN Fabrics can manage an OpenFabric mesh for you, and we moved to them in Phase 4. On a hand-built mesh, an FRR restart cut a node off for about 8 seconds. With a fabric left at PVE's defaults, an FRR restart cut a node off for 29 seconds. The reason: without lsp-gen-interval 1, FRR throttles link-state regeneration to 30 seconds. PVE's fabric options don't expose that, but PVE merges /etc/frr/frr.conf.local into the config it generates:

! /etc/frr/frr.conf.local
router openfabric mesh
 lsp-gen-interval 1
 max-lsp-lifetime 600
 lsp-refresh-interval 180
exit

With those timers, plus a hello interval of 1 second, CSNP of 2 seconds and a hello multiplier of 2 in the fabric settings, an FRR restart cost 2.5 seconds. DRBD stayed connected through it.

Tips for the cutover:

• Keep the fabric's router IDs equal to your existing loopback addresses, so corosync, LINSTOR and DRBD keep the same addresses.
• Write frr.conf.local right before the SDN apply. While the hand-built mesh is still running, the mere existence of that file makes any network reload regenerate FRR's config.
• Remove /etc/default/frr afterwards; PVE owns frr.conf from then on.
• Rehearse the rollback, on all three nodes at once.

Our production cutover took 3.5 minutes, with every node showing two OpenFabric neighbours afterwards.

Stable NIC names, staged before the upgrade#

New kernels and new systemd versions can rename NICs, and a renamed mesh port means no storage. We pinned names ourselves with systemd .link files that match on MAC address, such as mesh0/mesh1 for the mesh and up0/up1 for the uplinks.

• Stage them before the Phase 2 full-upgrade. The .link files have to be in the initramfs, and the upgrade rebuilds it. Then the first reboot both activates them and is the reboot you needed anyway.
• Rolling back means rebuilding the initramfs, for the same reason. We rehearsed that rollback: about two minutes.
• PVE 9 has pve-network-interface-pinning, but it only exists once you're on 9, and it doesn't update frr.conf.

Corosync and bonds#

• Corosync links: we made the mesh corosync's preferred link (priority 20) and kept management as the fallback (priority 10). The trade-off is that corosync now depends on FRR, so test failover both ways. Knet uses UDP port 5405 + the link number, which is handy for blocking one link in a test.
• Bonds: without MC-LAG on the switches we used active-backup bonds, with bond-primary-reselect failure. Automatic failback to the primary black-holed VM traffic for more than 10 seconds when the primary port came back.

HA surprises#

Proxmox HA mostly did what it should. But a full-outage upgrade puts it in states it doesn't see every day.

Parallel stop requests can get lost. We stopped the 17 HA guests with parallel API calls, and two of them never reached the HA manager. There was no task and no log entry, and our script discarded the errors of its parallel calls. Stop HA guests one at a time, or check each request's result and then wait for ha-manager status to show every one stopped.

HA fenced a node whose resource manager had just been restarted. This one surprised us the next morning. During the first reboot of one node:

• The package upgrade had restarted pve-ha-lrm about a minute before the reboot.
• With all its services stopped, the fresh LRM was still idle (wait_for_agent_lock) when the reboot began.
• It logged "reboot LRM, doing maintenance", but the HA manager saw the node go online → unknown instead of into maintenance. About 50 seconds later it marked the node fence.

No harm was done:

• the fence was acknowledged within a second, because nothing held the node's lock;
• there was no watchdog reset, and the shutdown was clean;
• the node's seven stopped HA guests were simply reassigned to the other nodes.

But it also sent two fence notifications to our alerting. If you reboot right after upgrading pve-ha-manager, put the node in maintenance first:

ha-manager crm-command node-maintenance enable pve1
# reboot, then:
ha-manager crm-command node-maintenance disable pve1

HA moves things around during a full shutdown.

• shutdown_policy=migrate moves even stopped HA services off a rebooting node.
• ha-rebalance-on-start moves guests while you're starting them.
• Between those two and the fence above, our HA guests ended up in quite different places from where we'd put them.

None of that matters while the guests are off. But with LINSTOR, a VM on a node without a local copy reads everything over the network. So after the night we live-migrated the four VMs that had ended up away from their disks back to nodes with a local copy. It took 24–43 seconds each.

A few more:

• After its services stop, an LRM can keep its watchdog armed for seven minutes or more.
• An API shutdown of an HA guest becomes an HA stop with a 300-second timeout.
• After a cold start of the whole cluster, the first HA start failed with "could not connect to any LINSTOR controller", because the controller wasn't up yet. HA's retry fixed it.

The night itself#

We drove the night from a tmux session on each node, with a second person watching every step. Each step was a script that logged to a file and ended with an ### END <step> rc=N line. The driver typed the step into the node's pane and waited for that line. There were five points where we stopped for an explicit go:

1. the shutdown;
2. the DRBD repair;
3. the checkpoint;
4. the first node's PVE 9 go/no-go;
5. the start-up.

Here's how it went, with T = the start of the outage:

The lab had projected T+2:52; we finished at T+3:06 inside a window of three and three-quarter hours. Production reboots took three and a half to four and a half minutes each, mostly firmware POST, which is exactly the kind of thing a nested lab can't tell you.

The surprises, all small:

• The lost HA stop requests. We finished the shutdown with a second script that stopped the two remaining guests one at a time. Our original stop script would have overwritten its own state file if we'd re-run it, so we didn't.
• chrony showed failed after the dist-upgrade. The new chrony had been restarted under the old systemd and couldn't notify it. It was fine after the reboot. Expect it, and don't chase it.
• One gate warned that drbdadm --dry-run adjust wanted 93 changes, seconds after another node had come back. A minute later it wanted none. The lab had shown the same thing.
• The HA fence and guest shuffling described above.

After the night:

• We tested a LINSTOR controller failover with drbd-reactorctl evict linstor_db. Another node took over in about 7 seconds.
• We live-migrated the four VMs that had ended up away from their disks back to nodes with a local copy.
• We scheduled a second full verify for the next morning, as systemd timers on the nodes.

The morning-after verify of all 93 disks on PVE 9 (with sha512) found 4 blocks on 2 disks where the replicas differed, down from 488 on 11 before the upgrade. We repaired them the same way, with the two VMs off, in 34 seconds.

Those same two disks had diverged before the upgrade too. Both belong to busy guests, so the guests' write load most likely caused the divergence, not the upgrade.

After the upgrade#

• Verify every disk again on the new versions. Use systemd timers, so it runs whether or not anyone's watching. Expect a few blocks on your busiest database VMs, and repair them in your next maintenance window.
• Keep an eye on the root snapshot with lvs, and drop it after a day or two once you're confident. A full snapshot becomes invalid, and it slows writes while it exists.
• Remove the old kernels after a couple of weeks.
• Clear leftover failed units with systemctl reset-failed.
• Check QEMU machine versions. VMs with an unpinned machine type moved to the newest version at their first cold start. Pinned ones, like our Windows guests, stayed where they were. That matters for some backup and replication tools, so check yours.
• Fix whatever surprised you, in the scripts, while you still remember. For us: serialize the HA stops and report each error, and put nodes into maintenance before any reboot that follows an HA package upgrade.

A checklist you can steal#

Weeks before#

• Build a lab with your exact package versions and configs, including your config mistakes.
• Put canary VMs on it that write continuously and report errors and stalls.
• Rehearse the full plan until it passes the first time, with production's own scripts.
• Decide: rolling or full outage; one night or two; the node order; the cut-off times.
• Check whether LINBIT's packages and DKMS build for the new kernel.

Days before#

• Run drbdadm verify on every disk, check every node's view, and plan the repairs.
• Get to DRBD 9.3.4 or later before the PVE 9 jump. If that can't happen first, make it Phase 2.
• Stage the scripts, configs and firmware, and pre-download the packages.
• Name the LINSTOR hold file *.pref, and confirm apt-cache policy honours it.
• Install the DRBD shutdown-order drop-in.
• Fix the post-up FRR restart and enable frr.service.

The night#

• Take a fresh backup and disable the scheduled job.
• Clear onboot on the guests; stop the HA guests one at a time and check each.
• Repair out-of-sync disks with everything off.
• Upgrade LINSTOR passive nodes first, and evict the controller before its node.
• Check the DKMS build for the new kernel before every reboot.
• Put HA nodes into maintenance before rebooting after an HA package upgrade.
• Gate between every node, and check every node's DRBD view each time.
• Stop for a go/no-go after the first PVE 9 node.
• Settle on one verify-alg everywhere, and not crct10dif.
• Set ha-rebalance-on-start to 0 during the start-up, then restore it.

After#

• Verify every disk again.
• Move VMs back to nodes with a local copy of their disks.
• Test a controller failover.
• Drop the root snapshot and the old kernels.

References#

• Proxmox: Upgrade from 8 to 9 · Full Mesh Network for Ceph Server (the mesh pattern we started from) · Roadmap and release notes
• LINBIT user guide: LINSTOR in Proxmox VE · LINSTOR administration (upgrades, VERSION_MISMATCH, the HA controller) · Upgrading DRBD · Throughput tuning
• DRBD ChangeLog: the 9.3.3 and 9.3.4 entries on out-of-sync bits and divergence
• linstor-proxmox CHANGELOG

Proxmox is a trademark of Proxmox Server Solutions GmbH. Dell is a trademark of Dell Inc. LINSTOR and DRBD are trademarks of LINBIT. Names and logos are used for identification only.