Usage

Hostdisk CSI can be used as a separate component that allows exposing manually formatted filesystem to kubernetes or together with Hostdisk Manager that allows for disk management kubernetes way using custom resources.

Usage with Hostdisk Manager

Display detected hostdisks

Hostdisk Manager will automatically detect disks connected to kubernetes nodes it's running on and will create Hostdisk resource for each of them. Hostdisk Manager supports seamless disk migration between nodes without requiring any additional configuration.

$ kubectl get hostdisk

NAME                                 NODE               DEVICE     SIZE          FSTYPE   STATE   AGE
0x6000c29011aaaad0389be38263e25a88   cluster06898-n01   /dev/sdb   512.00 GiB             Empty   3m19s
0x6000c291ed0bdb684594b0d103490829   cluster06898-n01   /dev/sdc   512.00 GiB             Empty   3m19s
0x6000c2937be9da9e2290110ab7258952   cluster06898-n02   /dev/sdb   512.00 GiB             Empty   3m19s
0x6000c299c3cc2bc861ecbbf273f76928   cluster06898-n02   /dev/sdc   512.00 GiB             Empty   3m19s
0x6000c29063e6f27e916637b5578826d0   cluster06898-n03   /dev/sdb   512.00 GiB             Empty   3m18s
0x6000c29112297be5906f0d7ea24fa28c   cluster06898-n03   /dev/sdc   512.00 GiB             Empty   3m18s

Info

Only devices that don't have a filesystem will be detected. To use pre-formatted disks, a Persistent Volume must be created manually.

Virtualization platforms

When deploying a Kubernetes cluster using virtualization platforms such as vSphere or Proxmox, ensure that your environment is configured to assign UUIDs to virtual machine disks. HostDisk Manager will only detect disks that have a valid UUID assigned.

In vSphere, this setting can be enabled through the virtual machine's configuration. Navigate to the VM's settings, then open the 'Advanced Parameters' tab. Add the parameter disk.EnableUUID and set its value to true. Ensure that the virtual machine is powered off before making this change.

Hostdisk formatting

Detected disks must be formatted with a supported filesystem before they can be used in Kubernetes workloads. Hostdisk Manager will then automatically provision a dedicated Persistent Volume for each formatted disk.

The desired filesystem can be defined through the Hostdisk specification. Filesystem can be set only once and can not be modified after setting. Supported filesystems: ext3, ext4, xfs.

kubectl

kubectl patch hostdisks.hostdisk.sarkan.io <HOSTDISK NAME> --type merge -p "{\"spec\":{\"format\":{\"filesystem\": {\"type\": \"xfs\"}}}}"

YAML

kubectl edit hostdisks "<HOSTDISK NAME>"

In spec.format.filesystem.type set name of the filesystem you want to use to format disk:
```
[...]
spec:
  format:
    filesystem:
      type: xfs
```

Multiple disk formatting

Use the provided script to format all detected disks with xfs. To use other filesystem type, edit the script accordingly.

kubectl get hostdisks.hostdisk.sarkan.io | grep -v NAME | cut -d " " -f 1 | while IFS= read -r DISK; do kubectl patch hostdisks.hostdisk.sarkan.io $DISK --type merge -p "{\"spec\":{\"format\":{\"filesystem\": {\"type\": \"xfs\"}}}}"; done

Persistent Volume creation

After disk is formatted using Hostdisk Manager it will automatically create Persistent Volume for this disk. Persistent Volume name will be equal to filesystem UUID of this disk. Hostdisk filesystem UUID and Persistent Volume can be displayed using kubectl. A full description of other possible Hostdisk states can be found in the Hostdisk lifecycle section.

$ kubectl get hostdisks -o wide
NAME                                 NODE                      DEVICE     SIZE        FSTYPE   FSUUID                                 STATE   AGE
0x6000c291b0af68729901c9df8c853655   node01.lab.indevops.com   /dev/sde   1.00 TiB    xfs      18d224bb-9c5f-45aa-9540-5b5f563d006c   Ready   3d

$ kubectl get persistentvolumes
NAME                                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      AGE
18d224bb-9c5f-45aa-9540-5b5f563d006c   1Ti        RWO            Retain           Available   3d

Info

If Persistent Volume associated to formatted disk is deleted Hostdisk Manager will create new Persistent Volume for this disk.

Changing disk filesystem

After filesystem format is set, it becomes immutable to prevent accidental reformatting and data loss. To change the filesystem you must delete corresponding hostdisk resource so it's no longer maintained by Hostdisk Manager and then manually wipe the filesystem on that disk. Once the new hostdisk resource has been created for the empty disk, you can format it as described in the previous section.

Wipe Hostdisk Manager managed disk:

Delete the Hostdisk resource and the associated Persistent Volume
On the node, manually wipe the filesystem on the selected disk

$ kubectl get hostdisks.hostdisk.sarkan.io 0x6000c291b0af68729901c9df8c853655 -o wide
NAME                                 NODE                      DEVICE     SIZE        FSTYPE   FSUUID                                 STATE   AGE
0x6000c291b0af68729901c9df8c853655   node01.lab.indevops.com   /dev/sde   1.00 TiB    xfs      18d224bb-9c5f-45aa-9540-5b5f563d006c   Ready   3d

$ kubectl delete hostdisk 0x6000c291b0af68729901c9df8c853655
$ kubectl delete pv 18d224bb-9c5f-45aa-9540-5b5f563d006c

$ ssh root@node01.lab.indevops.com wipefs -a /dev/sde

Tips

After disk is wiped it will be re-discovered within 60 seconds.

Manual usage (without hostdisk manager)

Node filesystem can be manually exposed by creating Persistent Volume with volumeHandle equal to filesystem UUID. It can be used if you don't want to use Hostdisk Manager or want to expose already formatted disk.

Warning

Manually created Persistent Volumes won't work after disk is moved to other node and will have to be recreated. It's recommended to use Hostdisk Manager.

Preparing disk

Before a disk can be exposed to Kubernetes, it must be formatted with a supported filesystem. The disk may already be formatted, or you can format it manually to the desired filesystem. We recommend formatting the disk with XFS without a partition (proceed with caution, you will lose all existing data on the device):
```
mkfs.xfs /dev/sdx
```

Use the following command to obtain the filesystem UUID:

$ blkid --cache-file /dev/null /dev/sdx
/dev/sdx: UUID="b8b64172-6b47-4da8-b467-581fecc993b2" BLOCK_SIZE="512" TYPE="xfs"

Use the following command to obtain the filesystem size and type:

$ lsblk -o PATH,SIZE,FSTYPE /dev/sdx
PATH     SIZE FSTYPE
/dev/sdx   1T xfs

You will also need name of the Kubernetes node where the filesystem is available. Usually it's the same as the node's hostname which can be obtained using following command:
```
$ hostnamectl --static
node01.lab.com
```

Creating Persistent Volume for a Preformatted Disk

Once you have the necessary disk details, including filesystem type, UUID, size, and the node name, you can define a Persistent Volume to expose the disk to Kubernetes.

Create a PersistentVolume to represent this filesystem in Kubernetes:

metadata.name - name of the Persistent Volume. Must be unique across the cluster.
spec.csi.fsType - name of the filesystem disk is formatted to.
spec.csi.volumeHandle - UUID of the formatted disk.
spec.capacity.storage - size of the disk.
spec.nodeAffinity.required.nodeSelectorTerms - node affinity specifying on which node disk is located. Required for proper workload scheduling.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: b8b64172-6b47-4da8-b467-581fecc993b2
spec:
  csi:
    driver: hostdisk.csi.sarkan.io
    fsType: xfs
    volumeHandle: b8b64172-6b47-4da8-b467-581fecc993b2
  capacity:
    storage: 1Ti
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node01.lab.com

Create a PersistentVolumeClaim for the PersistentVolume:

metadata.name - name of the Persistent Volume Claim. Must be unique in the namespace.
spec.volumeName - name of the Persistent Volume this claim should use.
spec.storageClassName - empty string must be explicitly set otherwise default StorageClass will be set.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: b8b64172-6b47-4da8-b467-581fecc993b2
  namespace: hostdisk-csi
spec:
  storageClassName: ""
  volumeName: b8b64172-6b47-4da8-b467-581fecc993b2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti