These are notes from Gabor, for LINBIT consumption only, containing tips/insights into LINSTOR's EXOS integration. Refining of these notes is needed, so feel free :)
https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#ch-exos
exos pic from UG
1 enclosure = 2 exos ctrl + 2 exos pools
both exos ctrls can access both exos pools
cabling automatically detected by linstor
exos REST: show ports
...
},
{
"status": "Up",
"target-id": "500c0ff3beed4000",
"fan-out": 1,
"configured-speed": "",
"actual-speed": "12Gb",
"media": "SAS",
"durable-id": "hostport_A0",
"health-reason": "",
"sas-port": [
{
"sas-disabled-lanes": 0,
"width": 4,
"sas-lanes-expected": 4,
"sas-active-lanes": 4,
"configured-topology": "Direct",
"object-name": "port-details",
"configured-topology-numeric": 0
}
],
"port-type-numeric": 8,
"controller": "A",
"health": "OK",
"controller-numeric": 1,
"actual-speed-numeric": 11,
"configured-speed-numeric": 255,
"object-name": "ports",
"health-numeric": 0,
"health-recommendation": "",
"port": "A0",
"status-numeric": 0,
"port-type": "SAS"
},
...
local: cat /sys/class/sas_device/end_device-*/sas_address
0x500c0ff3beed4000
0x500c0ff3beed4400
0x5001e676dcdde000
exos REST: show initiators
...
},
{
"profile": "Standard",
"host-key": "HU",
"host-port-bits-a": 1,
"host-port-bits-b": 0,
"durable-id": "I1",
"profile-numeric": 0,
"host-bus-type-numeric": 8,
"discovered": "Yes",
"object-name": "initiator",
"mapped": "No",
"host-bus-type": "SAS",
"host-id": "NOHOST",
"nickname": "",
"id": "500605b00c47e800"
},
...
local:
cat /sys/class/sas_phy/*/sas_address
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x500605b00c47e800
0x5001e676dcdde000
0x5001e676dcdde000
0x5001e676dcdde000
0x5001e676dcdde000
creating a volume:
1) create actual volume: REST: create volume pool ${poolname} size ${size} ${volume_name}
2) create a mapping to initiator: REST: map volume access rw initiator ${comma_separated_initiators} lun ${lun} port ${exos_port} ${volume_name}
3) scan for devices at initiator: shell: $ echo "- - -" > /sys/class/scsi_host/host${host_id}/scan
$ lsscsi
[0:0:0:0] enclosu SEAGATE 5525 E -
[0:0:0:4] disk SEAGATE 5525 E /dev/sdc
[0:0:1:0] enclosu SEAGATE 5525 E -
[0:0:1:4] disk SEAGATE 5525 E /dev/sdd
-> automatically multipath on "...:0:X" and "...:1:X"
$ multipathd show paths format "%i %d %m %t"
hcil dev multipath dm_st
1:0:0:0 sda [orphan] undef
2:0:0:0 sdb ST4000NC000-1FR168_Z3029CYW active
0:0:0:4 sdc 3600c0ff00029a5f5a93d9a6001000000 active
0:0:1:4 sdd 3600c0ff00029a5f5a93d9a6001000000 active
%i == hcil / hctl
%d == single device (sdb, sdc, ...)
%m == multipath device (i.e. 3600c0ff0003b7c784c7f206001000000)
%t == dm status (failed, undef, active ,..)
-> again, find multipath id by LUN, and use /dev/mapper/${multipath_id} as resulting device
linstor exos exec ${enclosure_name} <arbitrary exos api command>
finding local scsi host IDs:
1) $ lsscsi
[0:0:0:0] enclosu SEAGATE 5525 E -
[0:0:1:0] enclosu SEAGATE 5525 E -
2) look for "enclosu", remember the [x:x:$host:x]
using Exos REST api manually:
1) login:
export SESSION_KEY=$(curl --insecure -H 'datatype: json' https://172.16.16.12/api/login/0f71f5b3d3200ab9c7185c67ed25ee7716b91a3280a1666149aefcfbd4b04dc7 | jq -r '.status[0].response')
0f71f5b3d3200ab9c7185c67ed25ee7716b91a3280a1666149aefcfbd4b04dc7 => hex(sha256("manage_!manage"))
curl ... /api/login/...
{
"status": [
{
"object-name": "status",
"response-type": "Success",
"response-type-numeric": 0,
"response": "4837706964be95ec850fbbdf3fc5b0b2",
"return-code": 1,
"component-id": "",
"time-stamp": "2021-05-11 07:34:03",
"time-stamp-numeric": 1620718443
}
]
}
2) append the sessionKey and execute the REST commands. GET requests ONLY (no PUT, POST and not even DELETE!)
curl --insecure -H "datatype:json" -H "sessionKey: $SESSION_KEY" https://172.16.16.12/api/show/volumes | jq .
other useful APIs:
show volumes pools ${pool_name}
show pools
show volumes [${volume_name}]
create volume pool ${pool_name} size ${size}[${unit}] ${volume_name}
delete volumes ${volume_name}
show maps [${volume_name}]
map volume access rw initiator ${comma_separated_initiator_list} lun ${lun} port ${port} ${volume_name}
unmap volume [initiator ${comma_separated_initiator_list}] ${volume_name}
show controllers
show ports
show initiator
show system
show events both [last ${count}]
shared storage pools
* Linstor's Exos provider is based on shared storage pool feature. NOT the other way around
* you can create / test / use shared storage pools using iSCSI NVMe or having quemu configured shared disks
default setup:
+-----------------+ +-----------------+
| Node1 | | Node2 |
| +-------------+ | | +-------------+ |
| | Pool1 | | | | Pool2 | |
| | +---------+ | | | | +---------+ | |
| | | Rsc1 <====== DRBD ======> Rsc1 | | |
| | +---------+ | | | | +---------+ | |
| +-------------+ | | +-------------+ |
| | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | up | | | | up | |
| | nodeid 0 | | | | nodeid 1 | |
| | ... | | | | ... | |
| +-------------+ | | +-------------+ |
+-----------------+ +-----------------+
node1 and node2 having a shared storage pool:
+-----------------+ +-----------------+
| Node1 | | Node2 |
| +-----------------------------------------+ |
| | Shared Storage Pool | |
| | +-------------------------------------+ | |
| | | Rsc1 Rsc1 | | |
| | | ACTIVE INACTIVE | | |
| | +-------------------------------------+ | |
| +-----------------------------------------+ |
| | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | up | | | | | |
| | nodeid 0 | | | | down | |
| | ... | | | | | |
| +-------------+ | | +-------------+ |
+-----------------+ +-----------------+
DRBD® is allowed to be on top of Rsc1, but only Node1/Rsc1 OR Node2/Rsc1 can be active concurrently
moving active resource from node1 to node2:
=> linstor resource deactivate node1 rsc1
+-----------------+ +-----------------+
| Node1 | | Node2 |
| +-----------------------------------------+ |
| | Shared Storage Pool | |
| | +-------------------------------------+ | |
| | | Rsc1 Rsc1 | | |
| | | INACTIVE INACTIVE | | |
| | +-------------------------------------+ | |
| +-----------------------------------------+ |
| | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | | | | | | |
| | down | | | | down | |
| | | | | | | |
| +-------------+ | | +-------------+ |
+-----------------+ +-----------------+
=> linstor resource activate node2 rsc1
+-----------------+ +-----------------+
| Node1 | | Node2 |
| +-----------------------------------------+ |
| | Shared Storage Pool | |
| | +-------------------------------------+ | |
| | | Rsc1 Rsc1 | | |
| | | ACTIVE INACTIVE | | |
| | +-------------------------------------+ | |
| +-----------------------------------------+ |
| | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | | | | | up | |
| | down | | | | nodeid 0 | |
| | | | | | ... | |
| +-------------+ | | +-------------+ |
+-----------------+ +-----------------+
=> linstor resource activate node1 rsc1
ERROR:
Description:
Only one active resource per shared storage pool allowed
...
Additionally replicating to a third node...
+-----------------+ +-----------------+ +-----------------+
| Node1 | | Node2 | | Node3 |
| +-----------------------------------------+ | | +-------------+ |
| | Shared Storage Pool | | | | Pool3 | |
| | +-------------------------------------+ | | | | +---------+ | |
| | | Rsc1 Rsc1 <======= DRBD =======> Rsc1 | | |
| | | ACTIVE INACTIVE | | | | | | ACTIVE | | |
| | +-------------------------------------+ | | | | +---------+ | |
| +-----------------------------------------+ | | +-------------+ |
| | | | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | up | | | | | | | | up | |
| | nodeid 0 | | | | down | | | | nodeid 1 | |
| | ... | | | | | | | | ... | |
| +-------------+ | | +-------------+ | | +-------------+ |
+-----------------+ +-----------------+ +-----------------+
... which can also have a shared storage pool setup
+-----------------+ +-----------------+ +-----------------+ +-----------------+
| Node1 | | Node2 | | Node3 | | Node4 |
| +-----------------------------------------+ | | +-----------------------------------------+ |
| | Shared Storage Pool 12 | | | | Shared Storage Pool 34 | |
| | +-------------------------------------+ | | | | +-------------------------------------+ | |
| | | Rsc1 Rsc1 <======= DRBD =======> Rsc1 Rsc1 | | |
| | | ACTIVE INACTIVE | | | | | | ACTIVE INACTIVE | | |
| | +-------------------------------------+ | | | | +-------------------------------------+ | |
| +-----------------------------------------+ | | +-----------------------------------------+ |
| | | | | | | |
| +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ | | +-DRBD Rsc1/0-+ |
| | up | | | | | | | | up | | | | | |
| | nodeid 0 | | | | down | | | | nodeid 1 | | | | down | |
| | ... | | | | | | | | ... | | | | | |
| +-------------+ | | +-------------+ | | +-------------+ | | +-------------+ |
+-----------------+ +-----------------+ +-----------------+ +-----------------+
Recording of video/zoom presentation Gabor gave covering these notes.
https://drive.google.com/file/d/1PBkTKgfyFdbjc6eu8egunH9aGzvKn-X-/view?usp=sharing