Deploying in Kubernetes

This page describes to CSI driver developers how to deploy their driver onto a Kubernetes cluster.

Overview

There are three components plus the kubelet that enable CSI drivers to provide storage to Kubernetes. These components are sidecar containers which are responsible for communication with both Kubernetes and the CSI driver, making the appropriate CSI calls for their respectful Kubernetes events.

Sidecar Containers

sidecar-container

Sidecar containers manage Kubernetes events and make the appropriate calls to the CSI driver. These are the external attacher, external provisioner, and the driver registrar.

External Attacher

external-attacher is a sidecar container that watches Kubernetes VolumeAttachment objects and triggers CSI ControllerPublish and ControllerUnpublish operations against a driver endpoint. As of this writing, the external attacher does not support leader election and therefore there can be only one running per CSI driver. For more information please read Attaching and Detaching.

Note, even though this is called the external attacher, its function is to call the CSI API calls ControllerPublish and ControllerUnpublish. These calls most likely will occur in a node which is not the one that will mount the volume. For this reason, many CSI drivers do not support these calls, instead doing the attach/detach and mount/unmount both in the CSI NodePublish and NodeUnpublish calls done by the kubelet at the node which is supposed to mount.

External Provisioner

external-provisioner is a Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CSI CreateVolume and DeleteVolume operations against a driver endpoint. For more information please read Provisioning and Deleting.

External Snapshotter

external-snapshotter is a Sidecar container that watches Kubernetes VolumeSnapshot objects and triggers CSI CreateSnapshot and DeleteSnapshot operations against a driver endpoint. For more information please read Snapshot Design Proposal.

Driver Registrar

driver-registrar is a sidecar container that registers the CSI driver with kubelet, and adds the drivers custom NodeId to a label on the Kubernetes Node API Object. It does this by communicating with the Identity service on the CSI driver and also calling the CSI GetNodeId operation. The driver registrar must have the Kubernetes name for the node set through the environment variable KUBE_NODE_NAME as follows:

        - name: csi-driver-registrar
          imagePullPolicy: Always
          image: quay.io/k8scsi/driver-registrar:v0.2.0
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: socket-dir
              mountPath: /csi

Kubelet

kubelet

The Kubernetes kubelet runs on every node and is responsible for making the CSI calls NodePublish and NodeUnpublish. These calls mount and unmount the storage volume from the storage system, making it available to the Pod to consume. As shown in the external-attacher, most CSI drivers choose to implement both their attach/detach and mount/unmount calls in the NodePublish and NodeUnpublish calls. They do this because the kubelet makes the request on the node which is to consume the volume.

Mount point

The mount point used by the CSI driver must be set to Bidirectional. See the example below:

          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: mountpoint-dir
              mountPath: /var/lib/kubelet/pods
              mountPropagation: "Bidirectional"
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-hostpath
            type: DirectoryOrCreate
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/kubelet/pods
            type: Directory

RBAC Rules

Side car containers need the appropriate permissions to be able to access and manipulate Kubernetes objects. Here are the RBAC rules needed:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-hostpath-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["create", "delete", "get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update"]

Deploying

Deploying a CSI driver onto Kubernetes is highlighted in detail in Recommended Mechanism for Deploying CSI Drivers on Kubernetes.

Examples

  • Simple deployment example using a single pod for all components: see the hostpath example.
  • Full deployment example using a DaemonSet for the node plugin and StatefulSet for the controller plugin: check the NFS driver deployment files.

More information

For more information, please read CSI Volume Plugins in Kubernetes Design Doc.