Design

Hostdisk Container Storage Interface

Hostdisk Container Storage Interface Driver (CSIopen in new window) is responsible for exposing filesystems available on Kubernetes node hosts. CSI is a standard for exposing arbitrary storage systems to containerized workloads on Container Orchestration Systems like Kubernetes.

Feature	Hostdisk CSI
Access modes	`ReadWriteOnce`, `ReadWriteOncePod`
Dynamic Provisioning	No
Persistence (Beyond Pod Lifetime)	Persistent

Hostdisk CSI Driver doesn't implement CSI controller service and consists only of CSI Node Service (DaemonSet), it can be viewed using kubectl:

$ kubectl get pods -n hostdisk-csi 
NAME                      READY   STATUS    RESTARTS   AGE
hostdisk-csi-node-46p8m   2/2     Running   0          3d
hostdisk-csi-node-rb6cv   2/2     Running   0          3d
hostdisk-csi-node-v7srk   2/2     Running   0          3d

Hostdisk manager

Hostdisk Manager allows for autodiscovery of unformatted block devices, formatting them and creation of persistent volumes.

WARNING

For block device to be discovered it must have WWN (World Wide Nameopen in new window). So e.g loop devices won't be discovered.

Hostdisk Custom Resource

After empty disk is detected in the cluster Hostdisk custom resource representing this disk will be created.

metadata.name - WWN (World Wide Nameopen in new window) of the disk
status is automatically updated by Hostdisk Manager every 60 seconds or after disk formatting.

apiVersion: hostdisk.sarkan.io/v1alpha1
kind: Hostdisk
metadata:
  name: 0x6000c29070c91a35d17d7618f5d451da
spec: {}
status:
  device: /dev/sde
  fstype: ""
  node: node01.example.com
  sizeBytes: 16106127360
  sizeHumanReadable: 15.00 GiB
  state: Empty
  uuid: ""

WARNING

Manual creation of hostdisk for already formatted disk can result in wiping data on this disk and should be done with extreme caution.

Hostdisk events

All major disk actions (discovery, formatting, wiping, errors) will produce hostdisk related events that record information about lifecycle of disk.

You can display hostdisk events using kubectl describe or kubectl events command:

$ kubectl describe hostdisk 0x6000c29a87fa867833a8c497be7a687e

[...]
Events:
Type    Reason   Age   From              Message
----    ------   ----  ----              -------
Normal  Create   55s   hostdisk-manager  New disk detected (node: node01.example.com, path: /dev/sdd, size: 10.00 GiB)
Normal  Format   3s    hostdisk-manager  Disk formatted (uuid: b25a9495-e6d5-4097-985e-e7cf6039e84e, filesystem type: xfs)

kubectl events --for hostdisk/0x6000c29a87fa867833a8c497be7a687e

Hostdisk lifecycle

Hostdisk can be in one of following states:

Empty - A new disk discovered on node. Must not be formatted to be discovered.
Unmanaged - Disk was formatted not by hostdisk manager. If disk is in this state Hostdisk Manager won't perform any actions on that disk. It must be manually wiped to be returned to Empty state
PendingPVsDeletion - Waiting for PVs associated with this disk to be deleted.
DiskFormatting - Disk have format specified in spec and is waiting to be formatted.
DiskFormattingFailedBackOff - Disk can't be formatted to desired state. Formatting will be retried with exponential rate limiting.
Ready - Disk is formatted and have PV.
Missing - Disk can no longer be found in cluster.
Unknown - Unknown state. Is assigned after disk was missing and is reconnected to node. Cluster will try to find correct state based on hostdisk resource state and disk format.

State diagram of Hostdisk lifecycle is visible below:

Moving disk between nodes

Hostdisk Manager managed disk can be moved between cluster nodes with no additional operator activity.

If disk is disconnected from cluster node it will be marked as missing. After disk is connected to another node Hostdisk Manager will detect it and it'll update nodes affinity labels. Persistent Volumes and Persistent Volume Claims using this disk can stay the same, but pods utilizing these volumes will have to be redeployed.

In case of node failure all disks attached to it can be safely moved to another node.