The LINSTOR Operator for Kubernetes currently is capable of compiling DRBD® on RHEL, Ubuntu, and SLES based Kubernetes workers with the proper build dependencies installed. However, LKE uses Debian Bullseye based Kubernetes workers, which the LINSTOR Operator does not have an appropriate KernelModuleInjectionImage
for at the time of writing (Dec. 2022). Therefore, the DRBD kernel module must be compiled and installed directly on the LKE worker nodes before deploying the LINSTOR Operator, in addition to LINSTOR®'s generic prerequisites.
Prerequisites and Considerations for Deploying LINSTOR into LKE
Attaching Linode Volume(s) to Each LKE Worker
A generic requirement for deploying LINSTOR into Kubernetes is having an empty block device for LINSTOR to use as a storage pool for the volumes that LINSTOR will provision. You can attach many volumes to each worker. These volumes should remain unformatted and unused, or else LINSTOR will refuse to use them.
In the image below you will see an example of a Linode with an additional unused Linode volume attached.
This volume attaches to the Linode as /dev/sdb
which will be defined as a storage pool for LINSTOR in the Helm deployment step. Each node in your LKE cluster should be configured with its own additional Linode volume.
Building and Installing DRBD on the LKE Workers
SSH into each of your LKE workers and install the necessary packages for compiling and installing the DRBD kernel module:
# apt install linux-headers-$(uname -r) build-essential
Then, download the DRBD 9 kernel module source tar file of your choice from https://pkg.linbit.com/. At the time of writing, the latest currently recommended DRBD kernel module is version 9.1.12. Update the version number that is set in the BASH variable command that follows if a newer 9.1.x module has become available:
# DRBD_VERS=9.1.12 # curl -LO https://pkg.linbit.com//downloads/drbd/9/drbd-$DRBD_VERS.tar.gz # tar xvf drbd-$DRBD_VERS.tar.gz # cd drbd-$DRBD_VERS/ # make && make install # modprobe drbd
You can verify that the correct DRBD kernel module has been loaded by running the following command and verifying the first line that reads, version: 9.x.x
, matches the DRBD kernel module version you intended to build:
# cat /proc/drbd version: 9.1.12 (api:2/proto:86-121) GIT-hash: 86ec2326fef3aede9f4d46f52bfd35aac4d5eb7e build by root@lke83867-128388-639a05ada20c, 2022-12-14 20:31:50 Transports (api:18):
Disable the DRBD User Mode Helper on the LKE Workers
Finally, we will disable the DRBD user mode helper. This feature of the DRBD module enables running user configured commands on changes in DRBD state. When using DRBD within containers, it could confuse programs, such as the default drbdadm
, that expect to know about all configured DRBD resources. To prevent any issues, you have to set the DRBD module parameter usermode_helper
to disabled
on each LKE worker by using the following commands:
# echo -n disabled > /sys/module/drbd/parameters/usermode_helper # echo options drbd usermode_helper=disabled > /etc/modprobe.d/drbd.conf
Verify that the commands above succeeded and that the usermode_helper
is now set to disabled
and not drbdadm
:
# cat /sys/module/drbd/parameters/usermode_helper disabled
Deployment
With the prerequisites taken care of, all that's left is to deploy the LINSTOR Operator. Deployment of the LINSTOR Operator follows the instructions found in the LINSTOR User's Guide except that the following Helm options must be used because you have compiled and installed DRBD from source code:
operator: satelliteSet: kernelModuleInjectionImage: drbd.io/amd64/drbd9-jammy kernelModuleInjectionMode: DepsOnly
For completeness, the following commands include Helm options used to deploy LINSTOR into LKE where each LKE worker has a single additional Linode volume (/dev/sdb
) attached. This additional volume will be used by LINSTOR as a thin LVM backed storage pool named thin-lvm
:
# helm repo add linstor https://charts.linstor.io # helm repo update # kubectl create namespace linstor # kubectl create secret docker-registry drbdiocred -n linstor \ --docker-server=drbd.io --docker-username=$LBUSER \ --docker-email=$LBEMAIL --docker-password=$LBPASS # cat << EOF > linstor-op-vals.yaml operator: controller: dbConnectionURL: k8s satelliteSet: storagePools: lvmThinPools: - name: lvm-thin thinVolume: thinpool volumeGroup: devicePaths: - /dev/sdb kernelModuleInjectionImage: drbd.io/amd64/drbd9-jammy kernelModuleInjectionMode: DepsOnly etcd: enabled: false stork: enabled: false csi: enableTopology: true EOF # helm install -n linstor -f ./linstor-op-vals.yaml linstor-op linstor/linstor
Next, you can continue to configure LINSTOR's storage classes in Kubernetes as outlined in the LINSTOR User's Guide.
Written on 12/14/2022 - MDK
Reviewed on 12/15/2022 - MAT