Deploying in Kubernetes
This page describes to CSI driver developers how to deploy their driver onto a Kubernetes cluster.
There are three components plus the kubelet that enable CSI drivers to provide storage to Kubernetes. These components are sidecar containers which are responsible for communication with both Kubernetes and the CSI driver, making the appropriate CSI calls for their respectful Kubernetes events.
Sidecar containers manage Kubernetes events and make the appropriate calls to the CSI driver. These are the external attacher, external provisioner, and the driver registrar.
external-attacher is a sidecar container that watches Kubernetes VolumeAttachment objects and triggers CSI ControllerPublish and ControllerUnpublish operations against a driver endpoint. As of this writing, the external attacher does not support leader election and therefore there can be only one running per CSI driver. For more information please read Attaching and Detaching.
Note, even though this is called the external attacher, its function is to call the CSI API calls ControllerPublish and ControllerUnpublish. These calls most likely will occur in a node which is not the one that will mount the volume. For this reason, many CSI drivers do not support these calls, instead doing the attach/detach and mount/unmount both in the CSI NodePublish and NodeUnpublish calls done by the kubelet at the node which is supposed to mount.
external-provisioner is a Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CSI CreateVolume and DeleteVolume operations against a driver endpoint. For more information please read Provisioning and Deleting.
external-snapshotter is a Sidecar container that watches Kubernetes VolumeSnapshot objects and triggers CSI CreateSnapshot and DeleteSnapshot operations against a driver endpoint. For more information please read Snapshot Design Proposal.
driver-registrar is a sidecar container that registers the CSI driver with kubelet, and adds the drivers custom NodeId to a label on the Kubernetes Node API Object. It does this by communicating with the Identity service on the CSI driver and also calling the CSI GetNodeId operation. The driver registrar must have the Kubernetes name for the node set through the environment variable
KUBE_NODE_NAME as follows:
- name: csi-driver-registrar imagePullPolicy: Always image: quay.io/k8scsi/driver-registrar:v0.2.0 args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /csi/csi.sock - name: KUBE_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: socket-dir mountPath: /csi
The Kubernetes kubelet runs on every node and is responsible for making the CSI calls NodePublish and NodeUnpublish. These calls mount and unmount the storage volume from the storage system, making it available to the Pod to consume. As shown in the external-attacher, most CSI drivers choose to implement both their attach/detach and mount/unmount calls in the NodePublish and NodeUnpublish calls. They do this because the kubelet makes the request on the node which is to consume the volume.
The mount point used by the CSI driver must be set to Bidirectional. See the example below:
volumeMounts: - name: socket-dir mountPath: /csi - name: mountpoint-dir mountPath: /var/lib/kubelet/pods mountPropagation: "Bidirectional" volumes: - name: socket-dir hostPath: path: /var/lib/kubelet/plugins/csi-hostpath type: DirectoryOrCreate - name: mountpoint-dir hostPath: path: /var/lib/kubelet/pods type: Directory
Side car containers need the appropriate permissions to be able to access and manipulate Kubernetes objects. Here are the RBAC rules needed:
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-hostpath-role rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["create", "delete", "get", "list", "watch", "update"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "update"]
Deploying a CSI driver onto Kubernetes is highlighted in detail in Recommended Mechanism for Deploying CSI Drivers on Kubernetes.
- Simple deployment example using a single pod for all components: see the hostpath example.
- Full deployment example using a DaemonSet for the node plugin and StatefulSet for the controller plugin: check the NFS driver deployment files.
For more information, please read CSI Volume Plugins in Kubernetes Design Doc.