How to Use Device Mapper to Create Virtual Block Devices that Generate I/O Errors

Simulate I/O errors when an application attempts to access specific sectors of a disk. This can be particularly useful for testing how applications handle disk errors.

Overview

The Linux Device Mapper (dmsetup) is a versatile tool that allows users to create virtual block devices by mapping logical sectors to physical storage. One of its advanced features is the ability to create virtual devices that deliberately generate I/O errors when an application attempts to access specific sectors. This can be particularly useful for testing how applications handle disk errors.

This article will explain how to create such a device using the dmsetup create command with a practical example.

Example Scenario

Suppose you have a loop device (/dev/loop10) that you want to map into a new virtual block device, where certain sectors will cause I/O errors when accessed. You will create a new device called errdev0, which has three regions:

Region 1: Maps the first 261,144 sectors of /dev/loop10 directly.
Region 2: The next 5 sectors will be set to generate I/O errors.
Region 3: Maps the remaining sectors of /dev/loop10 after the error region.

Command Breakdown

dmsetup create errdev0 <<EOF
0 261144 linear /dev/loop10 0
261144 5 error
261149 3933155 linear /dev/loop10 261149
EOF

Steps and Explanation

Install Device Mapper Tools: Ensure the dmsetup command is available on your system. You can install it using your package manager if it’s not installed.
```
sudo apt-get install lvm2      # For Debian/Ubuntu-based systems
sudo dnf install device-mapper # For RHEL/AlmaLinux-based systems
```
Create a Loop Device (Optional): If you do not have a loop device (/dev/loop10 in this case), you can create one using an image file:
```
dd if=/dev/zero of=/var/lib/virt-disk-0.img bs=512 count=4194304  # Create a 2GB disk image
losetup /dev/loop10 /var/lib/virt-disk-0.img                      # Attach the loop device
```
TIP: Traditional disk based storage use a sector size of 512 bytes. Using bs=512 in the dd command above makes mapping regions of sectors easier because the file backed loop device will be count=4194304 sectors large. You could also use fdisk -l /dev/loop10 to get the sector count of the resulting loop device.
Create the “Bad” Block Device: Use the dmsetup create command to define a new virtual device (errdev0) with specific I/O error sectors:
```
dmsetup create errdev0 <<EOF
0 261144 linear /dev/loop10 0
261144 5 error
261149 3933155 linear /dev/loop10 261149
EOF
```
- 0 261144 linear /dev/loop10 0:
  - This maps the first 261,144 sectors of the new device (errdev0) directly to the first 261,144 sectors of /dev/loop10.
- 261144 5 error:
  - This defines a region of 5 sectors starting from sector 261,144 on errdev0 that will generate I/O errors when accessed. These errors simulate bad sectors.
- 261149 3933155 linear /dev/loop10 261149:
  - This maps the remaining sectors starting from 261,149 to the corresponding sectors on /dev/loop10.
Verify the New Device: After running the command, the new device errdev0 should appear under /dev/mapper/. You can verify its creation using:
```
ls /dev/mapper/errdev0
```
Test the Error Regions: You can now test accessing the device using tools like dd to observe the I/O errors:
```
dd if=/dev/mapper/errdev0 of=/dev/null bs=512 count=1 skip=261144
```
This should result in an error similar to:
```
dd: error reading '/dev/mapper/errdev0': Input/output error
```
Any attempt to read from the error region will fail, simulating a hardware failure.

Use Cases

Application Testing: Simulate bad sectors on a disk to test how software handles read/write errors.
Stress Testing: Verify the robustness of data recovery processes in storage systems.
Educational Purposes: Demonstrate how block device errors can affect applications.

Specific to LINBIT data replication software, you might want to test DRBD with different strategies for handling I/O errors using DRBD’s disk { on-io-error <strategy>; } option.

Tips for using errdev0 as a DRBD backing device to test I/O error handling:

Use the dmsetup create command in the previous section to create the errdev0 on only one node, the “bad node”. On other nodes in your DRBD cluster, you can create the errdev0 without any bad sectors using the following command:

dmsetup create errdev0 <<EOF
0 4194304 linear /dev/loop10 0
EOF

This will allow you to use identical DRBD configurations on all nodes in your test cluster.

An example configuration could look like this:

resource r0 {
  device /dev/drbd10;
  disk /dev/mapper/errdev0;
  meta-disk internal;

  disk {
    on-io-error pass_on; # possible strateg are `detatch`, `pass_on`, and `call-local-io-error`
  }

  handlers {
    #local-io-error <shell-script>; # handler called when `on-io-error call-local-io-error` is set
  }

  on linbit-0 {
    address   192.168.234.60:7770;
    node-id   0;
  }

  on linbit-1 {
    address   192.168.234.61:7770;
    node-id   1;
  }

  on linbit-2 {
    address   192.168.234.62:7770;
    node-id   2;
  }

  connection-mesh {
    hosts linbit-0 linbit-1 linbit-2;
  }
}

Skip the initial synchronization after creating metadata: On all nodes:
```
drbdadm create-md <res>
drbdadm up <res>
```
On the “bad node”:
```
drbdadm new-current-uuid --clear-bitmap <resource>/<volume>
```
If you do not skip the initial sync, DRBD will attempt to read that block during the initial synchronization, and detect the I/O error before you wanted it to.

Cleaning Up

To remove the virtual device and clean up the environment:

dmsetup remove errdev0
losetup -d /dev/loop10       # Detach the loop device (if used)
rm /var/lib/virt-disk-0.img  # Remove the disk image (if used)

Conclusion

Using the Device Mapper to create virtual block devices with intentional I/O errors is a powerful method for testing and validating software behavior under fault conditions. By following the steps outlined in this article, you can easily create and manage such devices for your testing needs.

- MDK 8/13/24