Resolving Version Mismatch Issues in LINSTOR

If you get a version mismatch message in LINSTOR, here are some common causes:

• A node is running DRBD® 8.x, while existing nodes are on DRBD 9.x. When using LINSTOR with DRBD, DRBD version 9.x is required.
• The LINSTOR satellite service version on a node is newer than the LINSTOR controller service and existing satellites.

When trying to resolve a version mismatch message, you might run into these related issues:

• Trying to roll back LINSTOR components results in an “unmet package dependencies” error.
• Nodes running different Linux distribution versions, for example, after an upgrade, can prevent access to package repositories that match the LINBIT® software package versions running on other nodes.

This article describes the steps that you can take to resolve version mismatch and related issues.

Sometimes, for example, when upgrading or when adding a LINSTOR satellite node to an existing LINSTOR® cluster, you might encounter the following version mismatch or related issues:

• LINSTOR version incompatibility: LINSTOR does not support rolling upgrades. LINSTOR controller and satellite services running in your cluster must have the same version, otherwise the controller will discard the satellite with a VERSION_MISMATCH error.
• Correct DRBD kernel module failing to load: For example, on a newly added node where you are using Dynamic Kernel Module Support (DKMS) to build the DRBD kernel, the system might load the DRBD 8.4 in-kernel module because the node is missing kernel headers that DKMS needs to compile a correct DRBD 9.x kernel module.

To resolve a version mismatch issue and bring cluster nodes into harmony, you can take the steps described in the following sections.

If you are using DKMS on your systems, install the correct kernel headers so that DKMS can build the proper DRBD 9.x module. You can do this in two ways, either by using a $(uname -r) package name construction when installing kernel headers for your system.

For example, on a Proxmox VE system:

apt install proxmox-headers-$(uname -r)

Alternatively and recommended, you can install the “default” kernel headers package. This way is recommended because having the default kernel headers package installed means that the system will automatically pull in the new headers when system upgrades happen.

For example, again on a Proxmox system:

apt install proxmox-default-headers

📝 NOTE: If you do not plan to install the latest kernel on your system, you would still need to install kernel headers that match your current running kernel by installing a -headers-$(uname -r) package, rather than a -default-headers package, before upgrading LINSTOR on a DKMS system.

After installing kernel headers, unload the DRBD kernel module, re-install the DRBD DKMS package, and then reload the DRBD kernel module.

For example, on a DEB-based system, enter the following commands:

rmmod drbd
apt install --reinstall drbd-dkms
modprobe drbd

Show information about the running DRBD kernel module and verify that the version number is what you are expecting:

cat /proc/drbd

For LINSTOR services to start and run successfully in your cluster, the DRBD version should show version: 9.x.x and match on all nodes.

Example output on an Ubuntu system:

version: 9.2.16 (api:2/proto:118-123)
[...]

Check the LINSTOR controller service version:

linstor controller version

On a LINSTOR satellite node that has a version mismatch error, list all available linstor-satellite package versions, after updating repository information.

On a DEB-based system:

apt update && apt-cache policy linstor-satellite

On an RPM-based system:

dnf info linstor-satellite

If the linstor-satellite package version needed to match the running LINSTOR controller service is not listed, you will need to add a repository that has a matching version.

To resolve package dependency issues, remove all LINSTOR components and reinstall with matching versions. Depending on the role of the node in your LINSTOR cluster (controller, satellite, or combined) you might not need to remove or install all of the packages shown in the following commands.

❗ IMPORTANT: Populate the LINSTORVERSION variable with the version number you need.

On a DEB-based system, for example:

LINSTORVERSION=1.32.3-1
apt remove -y linstor-satellite linstor-controller linstor-common
apt install -y linstor-common=$LINSTORVERSION
apt install -y linstor-satellite=$LINSTORVERSION
apt install -y linstor-controller=$LINSTORVERSION

On an RPM-based system, for example:

LINSTORVERSION=1.32.3-1
dnf remove -y linstor-satellite linstor-controller linstor-common
dnf install -y linstor-common-$LINSTORVERSION.noarch
dnf install -y linstor-satellite-$LINSTORVERSION.noarch
dnf install -y linstor-controller-$LINSTORVERSION.noarch

On DEB-based systems, LINSTOR services will restart automatically, after installing the packages. However, on RHEL-based systems, you need to manually restart the services. Depending on the role of the node in the cluster, enter the appropriate service restart command. On a combined node, enter both commands.

systemctl restart linstor-satellite

systemctl restart linstor-controller

After restarting LINSTOR services, enter another linstor node list command and verify that the output no longer shows a VERSION_MISMATCH error.

The following strategies can help you avoid version mismatch issues in your LINSTOR cluster.

• Avoid unintended upgrades on cluster nodes unless all nodes are upgraded together.
• Install the proxmox-default-headers package on all nodes to ensure DKMS can build DRBD modules for any installed kernel.
• If a node is upgraded and cannot be rebooted:
  • Verify that DRBD is built for the currently loaded kernel.
  • Use the following sequence (after downing DRBD resources) to unload and reload modules without rebooting:
    • Down all DRBD resources
    • Reload the DRBD kernel module (and possibly the DRBD transport module)
    • Verify the DRBD kernel module that you expect is loaded
    • Adjust all DRBD resource
• You can prepare for future upgrades by verifying that the dkms build process completes successfully for all installed kernels, not just the currently loaded one.

Written by MAT, 2025-11-06, with some initial article structure and content suggestions from ChatGPT and Gemini prompt responses.

Reviewed by JI & DJV, 2025-12-08.