This site documents how to develop, deploy, and test a Container Storage Interface (CSI) driver on Kubernetes.

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code.

The target audience for this site is third-party developers interested in developing CSI drivers for Kubernetes.

Kubernetes users interested in how to deploy or manage an existing CSI driver on Kubernetes should look at the documentation provided by the author of the CSI driver.

Kubernetes users interested in how to use a CSI driver should look at kubernetes.io documentation.

Kubernetes is as minimally prescriptive about packaging and deployment of a CSI Volume Driver as possible.

The only requirements are around how Kubernetes (master and node) components find and communicate with a CSI driver.

Specifically, the following is dictated by Kubernetes regarding CSI:

• Kubelet to CSI Driver Communication
  • Kubelet directly issues CSI calls (like NodeStageVolume, NodePublishVolume, etc.) to CSI drivers via a Unix Domain Socket to mount and unmount volumes.
  • Kubelet discovers CSI drivers (and the Unix Domain Socket to use to interact with a CSI driver) via the kubelet plugin registration mechanism.
  • Therefore, all CSI drivers deployed on Kubernetes MUST register themselves using the kubelet plugin registration mechanism on each supported node.
• Master to CSI Driver Communication
  • Kubernetes master components do not communicate directly (via a Unix Domain Socket or otherwise) with CSI drivers.
  • Kubernetes master components interact only with the Kubernetes API.
  • Therefore, CSI drivers that require operations that depend on the Kubernetes API (like volume create, volume attach, volume snapshot, etc.) MUST watch the Kubernetes API and trigger the appropriate CSI operations against it.

Because these requirements are minimally prescriptive, CSI driver developers are free to implement and deploy their drivers as they see fit.

That said, to ease development and deployment, the mechanism described below is recommended.

The Kubernetes development team has established a "Recommended Mechanism" for developing, deploying, and testing CSI Drivers on Kubernetes. It aims to reduce boilerplate code and simplify the overall process for CSI Driver developers.

This "Recommended Mechanism" makes use of the following components:

• Kubernetes CSI Sidecar Containers
• Kubernetes CSI objects
• CSI Driver Testing tools

To implement a CSI driver using this mechanism, a CSI driver developer should:

1. Create a containerized application implementing the Identity, Node, and optionally the Controller services described in the CSI specification (the CSI driver container).
   • See Developing CSI Driver for more information.
2. Unit test it using csi-sanity.
   • See Driver - Unit Testing for more information.
3. Define Kubernetes API YAML files that deploy the CSI driver container along with appropriate sidecar containers.
   • See Deploying in Kubernetes for more information.
4. Deploy the driver on a Kubernetes cluster and run end-to-end functional tests on it.
   • See Driver - Functional Testing

• Design Doc

The first step to creating a CSI driver is writing an application implementing the gRPC services described in the CSI specification

At a minimum, CSI drivers must implement the following CSI services:

• CSI Identity service
  • Enables callers (Kubernetes components and CSI sidecar containers) to identify the driver and what optional functionality it supports.
• CSI Node service
  • Only NodePublishVolume, NodeUnpublishVolume, and NodeGetCapabilities are required.
  • Required methods enable callers to make a volume available at a specified path and discover what optional functionality the driver supports.

All CSI services may be implemented in the same CSI driver application. The CSI driver application should be containerized to make it easy to deploy on Kubernetes. Once containerized, the CSI driver can be paired with CSI Sidecar Containers and deployed in node and/or controller mode as appropriate.

If your driver supports additional features, CSI "capabilities" can be used to advertise the optional methods/services it supports, for example:

• CONTROLLER_SERVICE (PluginCapability)
  • The entire CSI Controller service is optional. This capability indicates the driver implement one or more of the methods in the CSI Controller service.
• VOLUME_ACCESSIBILITY_CONSTRAINTS (PluginCapability)
  • This capability indicates the volumes for this driver may not be equally accessible from all nodes in the cluster, and that the driver will return additional topology related information that Kubernetes can use to schedule workloads more intelligently or influence where a volume will be provisioned.
• VolumeExpansion (PluginCapability)
  • This capability indicates the driver supports resizing (expanding) volumes after creation.
• CREATE_DELETE_VOLUME (ControllerServiceCapability)
  • This capability indicates the driver supports dynamic volume provisioning and deleting.
• PUBLISH_UNPUBLISH_VOLUME (ControllerServiceCapability)
  • This capability indicates the driver implements ControllerPublishVolume and ControllerUnpublishVolume -- operations that correspond to the Kubernetes volume attach/detach operations. This may, for example, result in a "volume attach" operation against the Google Cloud control plane to attach the specified volume to the specified node for the Google Cloud PD CSI Driver.
• CREATE_DELETE_SNAPSHOT (ControllerServiceCapability)
  • This capability indicates the driver supports provisioning volume snapshots and the ability to provision new volumes using those snapshots.
• CLONE_VOLUME (ControllerServiceCapability)
  • This capability indicates the driver supports cloning of volumes.
• STAGE_UNSTAGE_VOLUME (NodeServiceCapability)
  • This capability indicates the driver implements NodeStageVolume and NodeUnstageVolume -- operations that correspond to the Kubernetes volume device mount/unmount operations. This may, for example, be used to create a global (per node) volume mount of a block storage device.

This is an partial list, please see the CSI spec for a complete list of capabilities. Also see the Features section to understand how a feature integrates with Kubernetes.

This page summarizes major CSI changes made in each Kubernetes release. For details on individual features, visit the Features section.

• New beta features:
  • Volume cloning
  • Volume expansion
  • Ephemeral local volumes

• New features:
  • Volume capacity usage metrics
• New alpha features:
  • Volume cloning
  • Ephemeral local volumes
  • Resizing secrets

• csi.storage.k8s.io/v1alpha1 CSINodeInfo and CSIDriver CRDs are no longer supported.

• New beta features:
  • Topology
  • Raw block
  • Skip attach
  • Pod info on mount
• New alpha features:
  • Volume expansion
• New storage.k8s.io/v1beta1 CSINode and CSIDriver objects were introduced.

• CSI spec 0.2 and 0.3 are deprecated and support will be removed in Kubernetes 1.18.

• GA support added for CSI spec 1.0.

Kubelet device plugin registration is enabled by default, which requires CSI plugins to use driver-registrar:v0.3.0 to register with kubelet.

• New alpha features:
  • Snapshots
  • Topology
  • Skip attach
  • Pod info on mount
• New csi.storage.k8s.io/v1alpha1 CSINodeInfo and CSIDriver CRDs were introduced and have to be installed before deploying a CSI driver.

• Beta support added for CSI spec 0.3.
• New alpha features:
  • Raw block

• CSI spec 0.1 is no longer supported.

• Beta support added for CSI spec 0.2. This added optional NodeStageVolume and NodeUnstageVolume calls which map to Kubernetes MountDevice and UnmountDevice operations.

• Alpha support added for CSI spec 0.1.

Kubernetes CSI Sidecar Containers are a set of standard containers that aim to simplify the development and deployment of CSI Drivers on Kubernetes.

These containers contain common logic to watch the Kubernetes API, trigger appropriate operations against the “CSI volume driver” container, and update the Kubernetes API as appropriate.

The containers are intended to be bundled with third-party CSI driver containers and deployed together as pods.

The containers are developed and maintained by the Kubernetes Storage community.

Use of the containers is strictly optional, but highly recommended.

Benefits of these sidecar containers include:

• Reduction of "boilerplate" code.
  • CSI Driver developers do not have to worry about complicated, "Kubernetes specific" code.
• Separation of concerns.
  • Code that interacts with the Kubernetes API is isolated from (and in a different container then) the code that implements the CSI interface.

The Kubernetes development team maintains the following Kubernetes CSI Sidecar Containers:

• external-provisioner
• external-attacher
• external-snapshotter
• external-resizer
• node-driver-registrar
• cluster-driver-registrar (deprecated)
• livenessprobe

Every version of a sidecar has a minimum, maximum and recommended Kubernetes version that it is compatible with.

Minimum version specifies the lowest Kubernetes version where the sidecar will function with the most basic functionality, and no additional features added later. Generally, this aligns with the Kubernetes version where that CSI spec version was added.

Similarly, the max Kubernetes version generally aligns with when support for that CSI spec version was removed or if a particular Kubernetes API or feature was deprecated and removed.

It is important to note that any new features added to the sidecars may have dependencies on Kubernetes versions greater than the minimum Kubernetes version. The recommended Kubernetes version specifies the lowest Kubernetes version needed where all features of a sidecar will function correctly. Trying to use a new sidecar feature on a Kubernetes cluster below the recommended Kubernetes version may fail to function correctly. For that reason, it is encouraged to stay as close to the recommended Kubernetes version as possible.

For more details on which features are supported with which Kubernetes versions and their corresponding sidecars, please see each feature's individual page.

It is also important to note that alpha features are subject to break or be removed across Kubernetes and sidecar releases. There is no guarantee alpha features will continue to function if upgrading the Kubernetes cluster or upgrading a sidecar.

Git Repository: https://github.com/kubernetes-csi/external-attacher

Status: GA/Stable

Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page

The CSI external-attacher is a sidecar container that watches the Kubernetes API server for VolumeAttachment objects and triggers Controller[Publish|Unpublish]Volume operations against a CSI endpoint.

CSI drivers that require integrating with the Kubernetes volume attach/detach hooks should use this sidecar container, and advertise the CSI PUBLISH_UNPUBLISH_VOLUME controller capability.

For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-attacher/blob/master/README.md.

The CSI external-attacher is deployed as a controller. See deployment section for more details.

Git Repository: https://github.com/kubernetes-csi/external-provisioner

Status: GA/Stable

Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page

The CSI external-provisioner is a sidecar container that watches the Kubernetes API server for PersistentVolumeClaim objects.

It calls CreateVolume against the specified CSI endpoint to provision a new volume.

Volume provisioning is triggered by the creation of a new Kubernetes PersistentVolumeClaim object, if the PVC references a Kubernetes StorageClass, and the name in the provisioner field of the storage class matches the name returned by the specified CSI endpoint in the GetPluginInfo call.

Once a new volume is successfully provisioned, the sidecar container creates a Kubernetes PersistentVolume object to represent the volume.

The deletion of a PersistentVolumeClaim object bound to a PersistentVolume corresponding to this driver with a delete reclaim policy causes the sidecar container to trigger a DeleteVolume operation against the specified CSI endpoint to delete the volume. Once the volume is successfully deleted, the sidecar container also deletes the PersistentVolume object representing the volume.

The external-provisioner provides the ability to request a volume be pre-populated from a data source during provisioning. For more information on how data sources are handled see DataSources.

The CSI external-provisioner supports the Snapshot DataSource. If a Snapshot CRD is specified as a data source on a PVC object, the sidecar container fetches the information about the snapshot by fetching the SnapshotContent object and populates the data source field in the resulting CreateVolume call to indicate to the storage system that the new volume should be populated using the specified snapshot.

Cloning is also implemented by specifying a kind: of type PersistentVolumeClaim in the DataSource field of a Provision request. It's the responsbility of the external-provisioner to verify that the claim specified in the DataSource object exists, is in the same storage class as the volume being provisioned and that the claim is currently Bound.

When provisioning a new volume, the CSI external-provisioner sets the map<string, string> parameters field in the CSI CreateVolumeRequest call to the key/values specified in the StorageClass it is handling.

The CSI external-provisioner (v1.0.1+) also reserves the parameter keys prefixed with csi.storage.k8s.io/. Any keys prefixed with csi.storage.k8s.io/ are not passed to the CSI driver as an opaque parameter.

The following reserved StorageClass parameter keys trigger behavior in the CSI external-provisioner:

• csi.storage.k8s.io/provisioner-secret-name
• csi.storage.k8s.io/provisioner-secret-namespace
• csi.storage.k8s.io/controller-publish-secret-name
• csi.storage.k8s.io/controller-publish-secret-namespace
• csi.storage.k8s.io/node-stage-secret-name
• csi.storage.k8s.io/node-stage-secret-namespace
• csi.storage.k8s.io/node-publish-secret-name
• csi.storage.k8s.io/node-publish-secret-namespace
• csi.storage.k8s.io/fstype

If the PVC VolumeMode is set to Filesystem, and the value of csi.storage.k8s.io/fstype is specified, it is used to populate the FsType in CreateVolumeRequest.VolumeCapabilities[x].AccessType and the AccessType is set to Mount.

For more information on how secrets are handled see Secrets & Credentials.

Example StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gold-example-storage
provisioner: exampledriver.example.com
parameters:
  disk-type: ssd
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/provisioner-secret-name: mysecret
  csi.storage.k8s.io/provisioner-secret-namespace: mynamespace

CSI drivers that support dynamic volume provisioning should use this sidecar container, and advertise the CSI CREATE_DELETE_VOLUME controller capability.

For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-provisioner/blob/master/README.md.

The CSI external-provisioner is deployed as a controller. See deployment section for more details.

Git Repository: https://github.com/kubernetes-csi/external-resizer

Status: Beta starting with v0.3.0

Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page

The CSI external-resizer is a sidecar container that watches the Kubernetes API server for PersistentVolumeClaim object edits and triggers ControllerExpandVolume operations against a CSI endpoint if user requested more storage on PersistentVolumeClaim object.

CSI drivers that support Kubernetes volume expansion should use this sidecar container, and advertise the CSI VolumeExpansion plugin capability.

The CSI external-resizer is deployed as a controller. See deployment section for more details.

Git Repository: https://github.com/kubernetes-csi/external-snapshotter

Status: Alpha

Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page

The CSI external-snapshotter is a sidecar container that watches the Kubernetes API server for VolumeSnapshot and VolumeSnapshotContent CRD objects.

The creation of a new VolumeSnapshot object referencing a SnapshotClass CRD object corresponding to this driver causes the sidecar container to trigger a CreateSnapshot operation against the specified CSI endpoint to provision a new snapshot. When a new snapshot is successfully provisioned, the sidecar container creates a Kubernetes VolumeSnapshotContent object to represent the new snapshot.

The deletion of a VolumeSnapshot object bound to a VolumeSnapshotContent corresponding to this driver with a delete reclaim policy causes the sidecar container to trigger a DeleteSnapshot operation against the specified CSI endpoint to delete the snapshot. Once the snapshot is successfully deleted, the sidecar container also deletes the VolumeSnapshotContent object representing the snapshot.

For detailed information about volume snapshot and restore functionality, see Volume Snapshot & Restore.

CSI drivers that support provisioning volume snapshots and the ability to provision new volumes using those snapshots should use this sidecar container, and advertise the CSI CREATE_DELETE_SNAPSHOT controller capability.

For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md.

The CSI external-snapshotter is deployed as a controller. See deployment section for more details.

For an example deployment, see this example which deploys external-snapshotter and external-provisioner with the Hostpath CSI driver.

Git Repository: https://github.com/kubernetes-csi/livenessprobe

Status: GA/Stable

The CSI livenessprobe is a sidecar container that monitors the health of the CSI driver and reports it to Kubernetes via the Liveness Probe mechanism. This enables Kubernetes to automatically detect issues with the driver and restart the pod to try and fix the issue.

All CSI drivers should use the liveness probe to improve the availability of the driver while deployed on Kubernetes.

For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/livenessprobe/blob/master/README.md.

The CSI livenessprobe is deployed as part of controller and node deployments. See deployment section for more details.

Git Repository: https://github.com/kubernetes-csi/node-driver-registrar

Status: GA/Stable

The CSI node-driver-registrar is a sidecar container that fetches driver information (using NodeGetInfo) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.

Kubelet directly issues CSI NodeGetInfo, NodeStageVolume, and NodePublishVolume calls against CSI drivers. It uses the kubelet plugin registration mechanism to discover the unix domain socket to talk to the CSI driver. Therefore, all CSI drivers should use this sidecar container to register themselves with kubelet.

For detailed information (binary parameters, etc.), see the README of the relevant branch.

The CSI node-driver-registrar is deployed per node. See deployment section for more details.

This sidecar container was not updated since Kubernetes 1.13. As of Kubernetes 1.16, this side car container is officially deprecated.

The purpose of this side car container was to automatically register a CSIDriver object containing information about the driver with Kubernetes. Without this side car, developers and CSI driver vendors will now have to add a CSIDriver object in their installation manifest or any tool that installs their CSI driver.

Please see CSIDriver for more information.

Git Repository: https://github.com/kubernetes-csi/cluster-driver-registrar

Status: Alpha

The CSI cluster-driver-registrar is a sidecar container that registers a CSI Driver with a Kubernetes cluster by creating a CSIDriver Object which enables the driver to customize how Kubernetes interacts with it.

CSI drivers that use one of the following Kubernetes features should use this sidecar container:

• Skip Attach
  • For drivers that don't support ControllerPublishVolume, this indicates to Kubernetes to skip the attach operation and eliminates the need to deploy the external-attacher sidecar.
• Pod Info on Mount
  • This causes Kubernetes to pass metadata such as Pod name and namespace to the NodePublishVolume call.

If you are not using one of these features, this sidecar container (and the creation of the CSIDriver Object) is not required. However, it is still recommended, because the CSIDriver Object makes it easier for users to easily discover the CSI drivers installed on their clusters.

For detailed information (binary parameters, etc.), see the README of the relevant branch.

The CSI cluster-driver-registrar is deployed as a controller. See deployment section for more details.

Status: Beta

The Kubernetes API contains the following CSI specific objects:

• CSIDriver Object
• CSINode Object

Both are part of storage.k8s.io/v1beta1 API group.

The schema definition for the objects can be found here: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/storage/types.go

• Kubernetes 1.12 - 1.13: Alpha
• Kubernetes 1.14: Beta

The CSIDriver Kubernetes API object serves two purposes:

1. Simplify driver discovery

• If a CSI driver creates a CSIDriver object, Kubernetes users can easily discover the CSI Drivers installed on their cluster (simply by issuing kubectl get CSIDriver)

2. Customizing Kubernetes behavior

• Kubernetes has a default set of behaviors when dealing with CSI Drivers (for example, it calls the Attach/Detach operations by default). This object allows CSI drivers to specify how Kubernetes should interact with it.

Here is an example of a v1beta1 CSIDriver object:

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: mycsidriver.example.com
spec:
  attachRequired: true
  podInfoOnMount: true
  volumeLifecycleModes: # added in Kubernetes 1.16
  - Persistent
  - Ephemeral

There are three important fields:

• name
  • This should correspond to the full name of the CSI driver.
• attachRequired
  • Indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume method), and that Kubernetes should call attach and wait for any attach operation to complete before proceeding to mounting.
  • If a CSIDriver object does not exist for a given CSI Driver, the default is true -- meaning attach will be called.
  • If a CSIDriver object exists for a given CSI Driver, but this field is not specified, it also defaults to true -- meaning attach will be called.
  • For more information see Skip Attach.
• podInfoOnMount
  • Indicates this CSI volume driver requires additional pod information (like pod name, pod UID, etc.) during mount operations.
  • If value is not specified or false, pod information will not be passed on mount.
  • If value is set to true, Kubelet will pass pod information as volume_context in CSI NodePublishVolume calls:
    • "csi.storage.k8s.io/pod.name": pod.Name
    • "csi.storage.k8s.io/pod.namespace": pod.Namespace
    • "csi.storage.k8s.io/pod.uid": string(pod.UID)
  • For more information see Pod Info on Mount.
• volumeLifecycleModes
  • This field was added in Kubernetes 1.16 and cannot be set when using an older Kubernetes release.
  • It informs Kubernetes about the volume modes that are supported by the driver. This ensures that the driver is not used incorrectly by users. The default is Persistent, which is the normal PVC/PV mechanism. Ephemeral enables inline ephemeral volumes in addition (when both are listed) or instead of normal volumes (when it is the only entry in the list).

To install, a CSI driver's deployment manifest must contain a CSIDriver object as shown in the example above.

NOTE: The cluster-driver-registrar side-car which was used to create CSIDriver objects in Kubernetes 1.13 has been deprecated for Kubernetes 1.16. No cluster-driver-registrar has been released for Kubernetes 1.14 and later.

CSIDriver instance should exist for whole lifetime of all pods that use volumes provided by corresponding CSI driver, so Skip Attach and Pod Info on Mount features work correctly.

Using the CSIDriver object, it is now possible to query Kubernetes to get a list of registered drivers running in the cluster as shown below:

$> kubectl get csidrivers.storage.k8s.io
NAME                  CREATED AT
hostpath.csi.k8s.io   2019-09-13T09:58:43Z

Or get a more detailed view of your registered driver with:

$> kubectl describe csidrivers.storage.k8s.io
Name:         hostpath.csi.k8s.io
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"hostpath.csi.k8s.io"},"spec":{"podInfoOnMou...
API Version:  storage.k8s.io/v1beta1
Kind:         CSIDriver
Metadata:
  Creation Timestamp:  2019-09-13T09:58:43Z
  Resource Version:    341
  Self Link:           /apis/storage.k8s.io/v1beta1/csidrivers/hostpath.csi.k8s.io
  UID:                 1860f2a1-85f8-4357-a933-c45e54f0c8e0
Spec:
  Attach Required:    true
  Pod Info On Mount:  true
  Volume Lifecycle Modes:
    Persistent
    Ephemeral
Events:  <none>

During alpha development, the CSIDriver object was also defined as a Custom Resource Definition (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.

In the move from alpha to beta, the API Group for this object changed from csi.storage.k8s.io/v1alpha1 to storage.k8s.io/v1beta1.

There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.

In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of CSIDriver on these versions, do the following:

1. Ensure the feature gate is enabled via the following Kubernetes feature flag: --feature-gates=CSIDriverRegistry=true
2. Either ensure the CSIDriver CRD is automatically installed via the Kubernetes Storage CRD addon OR manually install the CSIDriver CRD on the Kubernetes cluster with the following command:

$> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csidriver.yaml

Kubernetes v1.14+, uses the same Kubernetes feature flag, but because the feature is beta, it is enabled by default. And since the API type (as of beta) is built in to the Kubernetes API, installation of the CRD is no longer required.

• Kubernetes 1.12 - 1.13: Alpha
• Kubernetes 1.14: Beta

CSI drivers generate node specific information. Instead of storing this in the Kubernetes Node API Object, a new CSI specific Kubernetes CSINode object was created.

It serves the following purposes:

1. Mapping Kubernetes node name to CSI Node name,

• The CSI GetNodeInfo call returns the name by which the storage system refers to a node. Kubernetes must use this name in future ControllerPublishVolume calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in the CSINode object for future reference.

2. Driver availability

• A way for kubelet to communicate to the kube-controller-manager and kubernetes scheduler whether the driver is available (registered) on the node or not.

3. Volume topology

• The CSI GetNodeInfo call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to do topology-aware provisioning (see PVC Volume Binding Modes for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall which Node label keys belong to a specific CSI driver, the kubelet stores the keys in the CSINode object for future reference.

Here is an example of a v1beta1 CSINode object:

apiVersion: storage.k8s.io/v1beta1
kind: CSINodeInfo
metadata:
  name: node1
spec:
  drivers:
  - name: mycsidriver.example.com
    nodeID: storageNodeID1
    topologyKeys: ['mycsidriver.example.com/regions', "mycsidriver.example.com/zones"]

What the fields mean:

• drivers - list of CSI drivers running on the node and their properties.
• name - the CSI driver that this object refers to.
• nodeID - the assigned identifier for the node as determined by the driver.
• topologyKeys - A list of topology keys assigned to the node as supported by the driver.

CSI drivers do not need to create the CSINode object directly. Instead they should use the node-driver-registrar sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the CSINode object on behalf of the the CSI driver.

The alpha object was called CSINodeInfo, whereas the beta object is called CSINode. The alpha CSINodeInfo object was also defined as a Custom Resource Definition (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.

In the move from alpha to beta, the API Group for this object changed from csi.storage.k8s.io/v1alpha1 to storage.k8s.io/v1beta1.

There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.

In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of CSINodeInfo on these versions, do the following:

1. Ensure the feature gate is enabled with --feature-gates=CSINodeInfo=true
2. Either ensure the CSIDriver CRD is automatically installed via the Kubernetes Storage CRD addon OR manually install the CSINodeInfo CRD on the Kubernetes cluster with the following command:

$> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csinodeinfo.yaml

Kubernetes v1.14+, uses the same Kubernetes feature flag, but because the feature is beta, it is enabled by default. And since the API type (as of beta) is built in to the Kubernetes API, installation of the CRD is no longer required.

The Kubernetes implementation of CSI has multiple sub-features. This section describes these sub-features, their status (although support for CSI in Kubernetes is GA/stable, support of sub-features moves independently so sub-features maybe alpha or beta), and how to integrate them in to your CSI Driver.

Some drivers may require a secret in order to complete operations.

If a CSI Driver requires secrets for a backend (a service account, for example), and this secret is required at the "per driver" granularity (not different "per CSI operation" or "per volume"), then the secret SHOULD be injected directly in to CSI driver pods via standard Kubernetes secret distribution mechanisms during deployment.

If a CSI Driver requires secrets "per CSI operation" or "per volume" or "per storage pool", the CSI spec allows secrets to be passed in for various CSI operations (including CreateVolumeRequest, ControllerPublishVolumeRequest, and more).

Cluster admins can populate such secrets by creating Kubernetes Secret objects and specifying the keys in the StorageClass or SnapshotClass objects.

The CSI sidecar containers facilitate the handling of secrets between Kubernetes and the CSI Driver. For more details see:

• StorageClass Secrets
• VolumeSnapshotClass Secrets

For reducing RBAC permissions as much as possible, secret rules are disabled in each sidecar repository by default.

Please add or update RBAC rules if secret is expected to use.

To set proper secret permission, uncomment related lines defined in rbac.yaml (e.g. external-provisioner/deploy/kubernetes/rbac.yaml)

CSI Drivers that accept secrets SHOULD handle this data carefully. It may contain sensitive information and MUST be treated as such (e.g. not logged).

To make it easier to handle secret fields (e.g. strip them from CSI protos when logging), the CSI spec defines a decorator (csi_secret) on all fields containing sensitive information. Any fields decorated with csi_secret MUST be treated as if they contain sensitive information (e.g. not logged, etc.).

The Kubernetes CSI development team also provides a GO lang package called protosanitizer that CSI driver developers may be used to remove values for all fields in a gRPC messages decorated with csi_secret. The library can be found in kubernetes-csi/csi-lib-utils/protosanitizer. The Kubernetes CSI Sidecar Containers and sample drivers use this library to ensure no sensitive information is logged.

The CSI external-provisioner sidecar container facilitates the handling of secrets for the following operations:

• CreateVolumeRequest
• DeleteVolumeRequest
• ControllerPublishVolumeRequest
• ControllerUnpublishVolumeRequest
• ControllerExpandVolumeRequest
• NodeStageVolumeRequest
• NodePublishVolumeRequest

CSI external-provisioner v1.0.1+ supports the following keys in StorageClass.parameters:

• csi.storage.k8s.io/provisioner-secret-name
• csi.storage.k8s.io/provisioner-secret-namespace
• csi.storage.k8s.io/controller-publish-secret-name
• csi.storage.k8s.io/controller-publish-secret-namespace
• csi.storage.k8s.io/node-stage-secret-name
• csi.storage.k8s.io/node-stage-secret-namespace
• csi.storage.k8s.io/node-publish-secret-name
• csi.storage.k8s.io/node-publish-secret-namespace

CSI external-provisioner v1.2.0+ adds support for the following keys in StorageClass.parameters:

• csi.storage.k8s.io/controller-expand-secret-name
• csi.storage.k8s.io/controller-expand-secret-namespace

Cluster admins can populate the secret fields for the operations listed above with data from Kubernetes Secret objects by specifying these keys in the StorageClass object.

In this example, the external-provisioner will fetch Kubernetes Secret object fast-storage-provision-key in the namespace pd-ssd-credentials and pass the credentials to the CSI driver named csi-driver.team.example.com in the CreateVolume CSI call.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast-storage
provisioner: csi-driver.team.example.com
parameters:
  type: pd-ssd
  csi.storage.k8s.io/provisioner-secret-name: fast-storage-provision-key
  csi.storage.k8s.io/provisioner-secret-namespace: pd-ssd-credentials

All volumes provisioned using this StorageClass use the same secret.

In this example, the external-provisioner will generate the name of the Kubernetes Secret object and namespace for the NodePublishVolume CSI call, based on the PVC namespace and annotations, at volume provision time.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast-storage
provisioner: csi-driver.team.example.com
parameters:
  type: pd-ssd
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['team.example.com/key']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}

This StorageClass will result in the creation of a PersistentVolume API object referencing a "node publish secret" in the same namespace as the PersistentVolumeClaim that triggered the provisioning and with a name specified as an annotation on the PersistentVolumeClaim. This could be used to give the creator of the PersistentVolumeClaim the ability to specify a secret containing a decryption key they have control over.

A drivers may support secret keys for multiple operations. In this case, you can provide secrets references for each operation:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast-storage-all
provisioner: csi-driver.team.example.com
parameters:
  type: pd-ssd
  csi.storage.k8s.io/provisioner-secret-name: ${pvc.name}
  csi.storage.k8s.io/provisioner-secret-namespace: ${pvc.namespace}-fast-storage
  csi.storage.k8s.io/node-publish-secret-name: ${pvc.name}-${pvc.annotations['team.example.com/key']}
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}-fast-storage
  

Details for each secret supported by the external-provisioner can be found below.

The CSI external-provisioner (v1.0.1+) looks for the following keys in StorageClass.parameters.

• csi.storage.k8s.io/provisioner-secret-name
• csi.storage.k8s.io/provisioner-secret-namespace

The values of both of these parameters, together, refer to the name and namespace of a Secret object in the Kubernetes API.

If specified, the CSI external-provisioner will attempt to fetch the secret before provisioning and deletion.

If the secret is retrieved successfully, the provisioner passes it to the CSI driver in the CreateVolumeRequest.secrets or DeleteVolumeRequest.secrets field.

If no such secret exists in the Kubernetes API, or the provisioner is unable to fetch it, the provision operation will fail.

Note, however, that the delete operation will continue even if the secret is not found (because, for example, the entire namespace containing the secret was deleted). In this case, if the driver requires a secret for deletion, then the volume and PV may need to be manually cleaned up.

The values of these parameters may be "templates". The external-provisioner will automatically resolve templates at volume provision time, as detailed below:

• csi.storage.k8s.io/provisioner-secret-name
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.
    • Support added in CSI external-provisioner v1.2.0+
  • ${pvc.name}
    • Replaced with the name of the PersistentVolumeClaim object that triggered provisioning.
    • Support added in CSI external-provisioner v1.2.0+
• csi.storage.k8s.io/provisioner-secret-namespace
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.

The CSI external-provisioner (v1.0.1+) looks for the following keys in StorageClass.parameters:

• csi.storage.k8s.io/controller-publish-secret-name
• csi.storage.k8s.io/controller-publish-secret-namespace

The values of both of these parameters, together, refer to the name and namespace of a Secret object in the Kubernetes API.

If specified, the CSI external-provisioner sets the CSIPersistentVolumeSource.ControllerPublishSecretRef field in the new PersistentVolume object to refer to this secret once provisioning is successful.

The CSI external-attacher then attempts to fetch the secret referenced by the CSIPersistentVolumeSource.ControllerPublishSecretRef, if specified, before an attach or detach operation.

If no such secret exists in the Kubernetes API, or the external-attacher is unable to fetch it, the attach or detach operation fails.

If the secret is retrieved successfully, the external-attacher passes it to the CSI driver in the ControllerPublishVolumeRequest.secrets or ControllerUnpublishVolumeRequest.secrets field.

The values of these parameters may be "templates". The external-provisioner will automatically resolve templates at volume provision time, as detailed below:

• csi.storage.k8s.io/controller-publish-secret-name
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.name}
    • Replaced with the name of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.annotations['<ANNOTATION_KEY>']} (e.g. ${pvc.annotations['example.com/key']})
    • Replaced with the value of the specified annotation from the PersistentVolumeClaim object that triggered provisioning
• csi.storage.k8s.io/controller-publish-secret-namespace
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.

The CSI external-provisioner (v1.0.1+) looks for the following keys in StorageClass.parameters:

• csi.storage.k8s.io/node-stage-secret-name
• csi.storage.k8s.io/node-stage-secret-namespace

The value of both parameters, together, refer to the name and namespace of the Secret object in the Kubernetes API.

If specified, the CSI external-provisioner sets the CSIPersistentVolumeSource.NodeStageSecretRef field in the new PersistentVolume object to refer to this secret once provisioning is successful.

The Kubernetes kubelet then attempts to fetch the secret referenced by the CSIPersistentVolumeSource.NodeStageSecretRef field, if specified, before a mount device operation.

If no such secret exists in the Kubernetes API, or the kubelet is unable to fetch it, the mount device operation fails.

If the secret is retrieved successfully, the kubelet passes it to the CSI driver in the NodeStageVolumeRequest.secrets field.

The values of these parameters may be "templates". The external-provisioner will automatically resolve templates at volume provision time, as detailed below:

• csi.storage.k8s.io/node-stage-secret-name
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.name}
    • Replaced with the name of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.annotations['<ANNOTATION_KEY>']} (e.g. ${pvc.annotations['example.com/key']})
    • Replaced with the value of the specified annotation from the PersistentVolumeClaim object that triggered provisioning
• csi.storage.k8s.io/node-stage-secret-namespace
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.

The CSI external-provisioner (v1.0.1+) looks for the following keys in StorageClass.parameters:

• csi.storage.k8s.io/node-publish-secret-name
• csi.storage.k8s.io/node-publish-secret-namespace

The value of both parameters, together, refer to the name and namespace of the Secret object in the Kubernetes API.

If specified, the CSI external-provisioner sets the CSIPersistentVolumeSource.NodePublishSecretRef field in the new PersistentVolume object to refer to this secret once provisioning is successful.

The Kubernetes kubelet, attempts to fetch the secret referenced by the CSIPersistentVolumeSource.NodePublishSecretRef field, if specified, before a mount operation.

If no such secret exists in the Kubernetes API, or the kubelet is unable to fetch it, the mount operation fails.

If the secret is retrieved successfully, the kubelet passes it to the CSI driver in the NodePublishVolumeRequest.secrets field.

The values of these parameters may be "templates". The external-provisioner will automatically resolve templates at volume provision time, as detailed below:

• csi.storage.k8s.io/node-publish-secret-name
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.name}
    • Replaced with the name of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.annotations['<ANNOTATION_KEY>']} (e.g. ${pvc.annotations['example.com/key']})
    • Replaced with the value of the specified annotation from the PersistentVolumeClaim object that triggered provisioning
• csi.storage.k8s.io/node-publish-secret-namespace
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.

The CSI external-provisioner (v1.2.0+) looks for the following keys in StorageClass.parameters:

• csi.storage.k8s.io/controller-expand-secret-name
• csi.storage.k8s.io/controller-expand-secret-namespace

The value of both parameters, together, refer to the name and namespace of the Secret object in the Kubernetes API.

If specified, the CSI external-provisioner sets the CSIPersistentVolumeSource.ControllerExpandSecretRef field in the new PersistentVolume object to refer to this secret once provisioning is successful.

The external-resizer (v0.2.0+), attempts to fetch the secret referenced by the CSIPersistentVolumeSource.ControllerExpandSecretRef field, if specified, before starting a volume resize (expand) operation.

If no such secret exists in the Kubernetes API, or the external-resizer is unable to fetch it, the resize (expand) operation fails.

If the secret is retrieved successfully, the external-resizer passes it to the CSI driver in the ControllerExpandVolumeRequest.secrets field.

The values of these parameters may be "templates". The external-provisioner will automatically resolve templates at volume provision time, as detailed below:

• csi.storage.k8s.io/controller-expand-secret-name
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.name}
    • Replaced with the name of the PersistentVolumeClaim object that triggered provisioning.
  • ${pvc.annotations['<ANNOTATION_KEY>']} (e.g. ${pvc.annotations['example.com/key']})
    • Replaced with the value of the specified annotation from the PersistentVolumeClaim object that triggered provisioning
• csi.storage.k8s.io/controller-expand-secret-namespace
  • ${pv.name}
    • Replaced with name of the PersistentVolume object being provisioned.
  • ${pvc.namespace}
    • Replaced with namespace of the PersistentVolumeClaim object that triggered provisioning.

The CSI external-snapshotter sidecar container facilitates the handling of secrets for the following operations:

• CreateSnapshotRequest
• DeleteSnapshotRequest

CSI external-snapshotter v1.0.1+ supports the following keys in VolumeSnapshotClass.parameters:

• csi.storage.k8s.io/snapshotter-secret-name
• csi.storage.k8s.io/snapshotter-secret-namespace

Cluster admins can populate the secret fields for the operations listed above with data from Kubernetes Secret objects by specifying these keys in the VolumeSnapshotClass object.

Details for each secret supported by the external-snapshotter can be found below.

CSI external-snapshotter v1.0.1+ looks for the following keys in VolumeSnapshotClass.parameters:

• csi.storage.k8s.io/snapshotter-secret-name
• csi.storage.k8s.io/snapshotter-secret-namespace

The values of both of these parameters, together, refer to the name and namespace of a Secret object in the Kubernetes API.

If specified, the CSI external-snapshotter will attempt to fetch the secret before creation and deletion.

If the secret is retrieved successfully, the snapshotter passes it to the CSI driver in the CreateSnapshotRequest.secrets or DeleteSnapshotRequest.secrets field.

If no such secret exists in the Kubernetes API, or the snapshotter is unable to fetch it, the create operation will fail.

Note, however, that the delete operation will continue even if the secret is not found (because, for example, the entire namespace containing the secret was deleted). In this case, if the driver requires a secret for deletion, then the volume and PV may need to be manually cleaned up.

The values of these parameters may be "templates". The external-snapshotter will automatically resolve templates at snapshot create time, as detailed below:

• csi.storage.k8s.io/snapshotter-secret-name
  • ${volumesnapshotcontent.name}
    • Replaced with name of the VolumeSnapshotContent object being created.
  • ${volumesnapshot.namespace}
    • Replaced with namespace of the VolumeSnapshot object that triggered creation.
  • ${volumesnapshot.name}
    • Replaced with the name of the VolumeSnapshot object that triggered creation.
• csi.storage.k8s.io/snapshotter-secret-namespace
  • ${volumesnapshotcontent.name}
    • Replaced with name of the VolumeSnapshotContent object being created.
  • ${volumesnapshot.namespace}
    • Replaced with namespace of the VolumeSnapshot object that triggered creation.

Some storage systems expose volumes that are not equally accessible by all nodes in a Kubernetes cluster. Instead volumes may be constrained to some subset of node(s) in the cluster. The cluster may be segmented into, for example, “racks” or “regions” and “zones” or some other grouping, and a given volume may be accessible only from one of those groups.

To enable orchestration systems, like Kubernetes, to work well with storage systems which expose volumes that are not equally accessible by all nodes, the CSI spec enables:

1. Ability for a CSI Driver to opaquely specify where a particular node exists (e.g. "node A" is in "zone 1").
2. Ability for Kubernetes (users or components) to influence where a volume is provisioned (e.g. provision new volume in either "zone 1" or "zone 2").
3. Ability for a CSI Driver to opaquely specify where a particular volume exists (e.g. "volume X" is accessible by all nodes in "zone 1" and "zone 2").

Kubernetes and the external-provisioner use these abilities to make intelligent scheduling and provisioning decisions (that Kubernetes can both influence and act on topology information for each volume),

To support topology in a CSI driver, the following must be implemented:

• The PluginCapability must support VOLUME_ACCESSIBILITY_CONTRAINTS.
• The plugin must fill in accessible_topology in NodeGetInfoResponse. This information will be used to populate the Kubernetes CSINode object and add the topology labels to the Node object.
• During CreateVolume, the topology information will get passed in through CreateVolumeRequest.accessibility_requirements.

In the StorageClass object, both volumeBindingMode values of Immediate and WaitForFirstConsumer are supported.

• If Immediate is set, then the external-provisioner will pass in all available topologies in the cluster for the driver.
• If WaitForFirstConsumer is set, then the external-provisioner will wait for the scheduler to pick a node. The topology of that selected node will then be set as the first entry in CreateVolumeRequest.accessibility_requirements.preferred. All remaining topologies are still included in the requisite and preferred fields to support storage systems that span across multiple topologies.

The topology feature requires the external-provisioner sidecar with the Topology feature gate enabled:

--feature-gates=Topology=true

In the Kubernetes cluster the CSINodeInfo feature must be enabled on both Kubernetes master and nodes (refer to the CSINode Object section for more info):

--feature-gates=CSINodeInfo=true

In order to fully function properly, all Kubernetes master and nodes must be on at least Kubernetes 1.14. If a selected node is on a lower version, topology is ignored and not passed to the driver during CreateVolume.

The alpha feature in the external-provisioner is not compatible across Kubernetes versions. In addition, Kubernetes master and node version skew and upgrades are not supported.

The CSINodeInfo, VolumeScheduling, and KubeletPluginsWatcher feature gates must be enabled on both Kubernetes master and nodes.

The CSINodeInfo CRDs also have to be manually installed in the cluster.

Note that a storage system may also have an "internal topology" different from (independent of) the topology of the cluster where workloads are scheduled. Meaning volumes exposed by the storage system are equally accessible by all nodes in the Kubernetes cluster, but the storage system has some internal topology that may influence, for example, the performance of a volume from a given node.

CSI does not currently expose a first class mechanism to influence such storage system internal topology on provisioning. Therefore, Kubernetes can not programmatically influence such topology. However, a CSI Driver may expose the ability to specify internal storage topology during volume provisioning using an opaque parameter in the CreateVolume CSI call (CSI enables CSI Drivers to expose an arbitrary set of configuration options during dynamic provisioning by allowing opaque parameters to be passed from cluster admins to the storage plugins) -- this would enable cluster admins to be able to control the storage system internal topology during provisioning.

This page documents how to implement raw block volume support to a CSI Driver.

A block volume is a volume that will appear as a block device inside the container. A mounted (file) volume is volume that will be mounted using a specified file system and appear as a directory inside the container.

The CSI spec supports both block and mounted (file) volumes.

CSI doesn't provide a capability query for block volumes, so COs will simply pass through requests for block volume creation to CSI plugins, and plugins are allowed to fail with the InvalidArgument GRPC error code if they don't support block volumes. Kubernetes doesn't make any assumptions about which CSI plugins support blocks and which don't, so users have to know if any given storage class is capable of creating block volumes.

The difference between a request for a mounted (file) volume and a block volume is the VolumeCapabilities field of the request. Note that this field is an array and the created volume must support ALL of the capabilities requested, or else return an error. If the AccessType method of a VolumeCapability VolumeCapability_Block, then the capability is requesting a raw block volume. Unlike mount volumes, block volumes don't have any specific capabilities that need to be validated, although access modes still apply.

Block volumes are much more likely to support multi-node flavors of VolumeCapability_AccessMode_Mode than mount volumes, because there's no file system state stored on the node side that creates any technical impediments to multi-attaching block volumes. While there may still be good reasons to prevent multi-attaching block volumes, and there may be implementations that are not capable of supporting multi-attach, you should think carefully about what makes sense for your driver.

CSI plugins that support both mount and block volumes must be sure to check the capabilities of all CSI RPC requests and ensure that the capability of the request matches the capability of the volume, to avoid trying to do file-system-related things to block volumes and block-related things to file system volumes. The following RPCs specify capabilities that must be validated:

• CreateVolume() (multiple capabilities)
• ControllerPublishVolume()
• ValidateVolumeCapabilities() (multiple capabilities)
• GetCapacity() (see below)
• NodeStageVolume()
• NodePublishVolume()

Also, CSI plugins that implement the optional GetCapacity() RPC should note that that RPC includes capabilities too, and if the capacity for mount volumes is not the same as the capacity for block volumes, that needs to be handled in the implementation of that RPC.

Q: Can CSI plugins support only block volumes and not mount volumes? A: Yes! This is just the reverse case of supporting mount volumes only. Plugins may return InvalidArgument for any creation request with an AccessType of VolumeCapability_Mount.

The main difference between block volumes and mount volumes is the expected result of the NodePublish(). For mount volumes, the CO expects the result to be a mounted directory, at TargetPath. For block volumes, the CO expects there to be a device file at TargetPath. The device file can by a bind-mounted device from the hosts /dev file system, or it can be a device node created at that location using mknod().

It's desirable but not required to expose an unfiltered device node. For example, CSI plugins based on technologies that implement SCSI protocols should expect that pods consuming the block volumes they create may want to send SCSI commands to the device. This is something that should "just work" by default (subject to container capabilities) so CSI plugins should avoid anything that would break this kind of use case. The only hard requirement is that the device implements block reading/writing however.

For plugins with the RPC_STAGE_UNSTAGE_VOLUME capability, the CO doesn't care exactly what is placed at the StagingTargetPath, but it's worth noting that some CSI RPCs are allowed to pass the plugin either a staging path or a publish path, so it's important to think carefully about how NodeStageVolume() is implemented, knowing that either path could get used by the CO to refer to the volume later on. This is made more challenging because the CSI spec says that StagingTargetPath is always a directory even for block volumes.

The raw block feature requires the external-provisioner and external-attacher sidecars to be deployed.

The BlockVolume and CSIBlockVolume feature gates need to be enabled on all Kubernetes masters and nodes.

--feature-gates=BlockVolume=true,CSIBlockVolume=true...

• TODO: detail how Kubernetes API raw block fields get mapped to CSI methods/fields.

Volume drivers, like NFS, for example, have no concept of an attach (ControllerPublishVolume). However, Kubernetes always executes Attach and Detach operations even if the CSI driver does not implement an attach operation (i.e. even if the CSI Driver does not implement a ControllerPublishVolume call).

This was problematic because it meant all CSI drivers had to handle Kubernetes attachment. CSI Drivers that did not implement the PUBLISH_UNPUBLISH_VOLUME controller capability could work around this by deploying an external-attacher and the external-attacher would responds to Kubernetes attach operations and simply do a noop (because the CSI driver did not advertise the PUBLISH_UNPUBLISH_VOLUME controller capability).

Although the workaround works, it adds an unnecessary operation (round-trip) in the preparation of a volume for a container, and requires CSI Drivers to deploy an unnecessary sidecar container (external-attacher).

The CSIDriver Object enables CSI Drivers to specify how Kubernetes should interact with it.

Specifically the attachRequired field instructs Kubernetes to skip any attach operation altogether.

For example, the existence of the following object would cause Kubernetes to skip attach operations for all CSI Driver testcsidriver.example.com volumes.

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: testcsidriver.example.com
spec:
  attachRequired: false

CSIDriver object should be manually included in the driver deployment manifests.

Previously, the cluster-driver-registrar sidecar container could be deployed to automatically create the object. Once the flags to this container are configured correctly, it will automatically create a CSIDriver Object when it starts with the correct fields set.

In alpha, this feature was enabled via the CSIDriver Object CRD.

apiVersion: csi.storage.k8s.io/v1alpha1
kind: CSIDriver
metadata:
....

CSI avoids encoding Kubernetes specific information in to the specification, since it aims to support multiple orchestration systems (beyond just Kubernetes).

This can be problematic because some CSI drivers require information about the workload (e.g. which pod is referencing this volume), and CSI does not provide this information natively to drivers.

The CSIDriver Object enables CSI Drivers to specify how Kubernetes should interact with it.

Specifically the podInfoOnMount field instructs Kubernetes that the CSI driver requires additional pod information (like podName, podUID, etc.) during mount operations.

For example, the existence of the following object would cause Kubernetes to add pod information at mount time to the NodePublishVolumeRequest.publish_context map.

apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
  name: testcsidriver.example.com
spec:
  podInfoOnMount: true

If the podInfoOnMount field is set to true, during mount, Kubelet will add the following key/values to the publish_context field in the CSI NodePublishVolumeRequest:

• csi.storage.k8s.io/pod.name: {pod.Name}
• csi.storage.k8s.io/pod.namespace: {pod.Namespace}
• csi.storage.k8s.io/pod.uid: {pod.UID}

The CSIDriver object should be manually included in the driver manifests.

Previously, the cluster-driver-registrar sidecar container could be used to create the object. Once the flags to this container are configured correctly, it will automatically create a CSIDriver Object when it starts with the correct fields set.

In alpha, this feature was enabled by setting the podInfoOnMountVersion field in the CSIDriver Object CRD to to v1.

apiVersion: csi.storage.k8s.io/v1alpha1
kind: CSIDriver
metadata:
  name: testcsidriver.example.com
spec:
  podInfoOnMountVersion: v1

A storage provider that allows volume expansion after creation, may choose to implement volume expansion either via a control-plane CSI RPC call or via node CSI RPC call or both as a two step process.

To implement volume expansion the CSI driver MUST:

1. Implement VolumeExpansion plugin capability.
2. Implement EXPAND_VOLUME controller capability or implement EXPAND_VOLUME node capability or both.

ControllerExpandVolume RPC call can be made when volume is ONLINE or OFFLINE depending on VolumeExpansion plugin capability. Where ONLINE and OFFLINE means:

1. ONLINE : Volume is currently published or available on a node.
2. OFFLINE : Volume is currently not published or available on a node.

NodeExpandVolume RPC call on the other hand - always requires volume to be published or staged on a node (and hence ONLINE). For block storage file systems - NodeExpandVolume is typically used for expanding the file system on the node, but it can be also used to perform other volume expansion related housekeeping operations on the node.

For details, see the CSI spec.

The Kubernetes CSI development team maintains external-resizer Kubernetes CSI Sidecar Containers. This sidecar container implements the logic for watching the Kubernetes API for Persistent Volume claim edits and issuing ControllerExpandVolume RPC call against a CSI endpoint and updating PersistentVolume object to reflect new size.

This sidecar is needed even if CSI driver does not have EXPAND_VOLUME controller capability, in this case it performs a NO-OP expansion and updates PersistentVolume object. NodeExpandVolume is always called by Kubelet on the node.

For more details, see external-resizer.

To expand a volume if permitted by the storage class, users just need to edit the persistent volume claim object and request more storage. Volume expansion for CSI volumes is an alpha feature and hence also must be explicitly enabled via feature gate:

--feature-gates=ExpandCSIVolumes=true

To enable online expansion of CSI persistent volumes, we also need to enable online expansion feature gate:

--feature-gates=ExpandInUsePersistentVolumes=true

It is expected that external-resizer and kubelet will add appropriate events and conditions to persistent volume claim object to indicate progress of volume expansion operation.

When creating a new PersistentVolumeClaim, the Kubernetes API provides a PersistentVolumeClaim.DataSource parameter. This parameter is used to specify the CSI CreateVolumeRequest.VolumeContentSource option for CSI Provisioners. The VolumeContentSource parameter instructs the CSI plugin to pre-populate the volume being provisioned with data from the specified source.

If a DataSource is specified in the CreateVolume call to the CSI external provisioner, the external provisioner will fetch the specified resource and pass the appropriate object id to the plugin.

Currently there are two types of PersistentVolumeClaim.DataSource objects that are supported:

1. VolumeSnapshot
2. PersistentVolumeClaim (Cloning)

A Clone is defined as a duplicate of an existing Kubernetes Volume. For more information on cloning in Kubernetes see the concepts doc for Volume Cloning. A storage provider that allows volume cloning as a create feature, may choose to implement volume cloning via a control-plan CSI RPC call.

For details regarding the kuberentes API for volume cloning, please see kubernetes concepts.

To implement volume cloning the CSI driver MUST:

1. Implement checks for csi.CreateVolumeRequest.VolumeContentSource in the plugin's CreateVolume function implementation.
2. Implement CLONE_VOLUME controller capability.

It is the responsibility of the storage plugin to either implement an expansion after clone if a provision request size is greater than the source, or allow the external-resizer to handle it. In the case that the plugin does not support resize capability and it does not have the capability to create a clone that is greater in size than the specified source volume, then the provision request should result in a failure.

The Kubernetes CSI development team maintains the external-provisioner which is responsible for detecting requests for a PVC DataSource and providing that information to the plugin via the csi.CreateVolumeRequest. It's up to the plugin to check the csi.CreateVolumeRequest for a VolumeContentSource entry in the CreateVolumeRequest object.

There are no additional side-cars or add on components required.

In Kubernetes 1.15 this feature was alpha status and required enabling the appropriate feature gate:

--feature-gates=VolumePVCDataSource=true

A trivial example implementation can be found in the csi-hostpath plugin in its implementation of CreateVolume.

Many storage systems provide the ability to create a "snapshot" of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore the existing volume to a previous state (represented by the snapshot).

Kubernetes CSI currently enables CSI Drivers to expose the following functionality via the Kubernetes API:

1. Creation and deletion of volume snapshots via Kubernetes native API.
2. Creation of new volumes pre-populated with the data from a snapshot via Kubernetes dynamic volume provisioning.

To implement the snapshot feature, a CSI driver MUST:

• Implement the CREATE_DELETE_SNAPSHOT and, optionally, the LIST_SNAPSHOTS controller capabilities
• Implement CreateSnapshot, DeleteSnapshot, and, optionally, the ListSnapshots, controller RPCs.

For details, see the CSI spec.

The Kubernetes CSI development team maintains the external-snapshotter Kubernetes CSI Sidecar Containers. This sidecar container implements the logic for watching the Kubernetes API for snapshot objects and issuing the appropriate CSI snapshot calls against a CSI endpoint. For more details, see external-snapshotter documentation.

Similar to the API for managing Kubernetes Persistent Volumes, the Kubernetes Volume Snapshots introduce three new API objects for managing snapshots: VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass. See Kubernetes Snapshot documentation for more details.

Unlike the core Kubernetes Persistent Volume objects, these Snapshot objects are defined as Custom Resource Definitions (CRDs). This is because the Kubernetes project is moving away from having resource types pre-defined in the API server. This allows the API server to be reused for projects other than Kubernetes, and consumers (like Kubernetes) simply install the resource types they require as CRDs. Because the Snapshot API types are not built in to Kubernetes, they must be installed prior to use.

The CRDs are automatically deployed by the CSI external-snapshotter sidecar.

The schema definition for the custom resources (CRs) can be found here: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go

In addition to these new CRD objects, a new, alpha DataSource field has been added to the PersistentVolumeClaim object. This new field enables dynamic provisioning of new volumes that are automatically pre-populated with data from an existing snapshot.

Since volume snapshot is an alpha feature in Kubernetes v1.12, you need to enable a new alpha feature gate called VolumeSnapshotDataSource in the Kubernetes master.

--feature-gates=VolumeSnapshotDataSource=true

Use the following example yaml files to test the snapshot feature.

Create a StorageClass:

kubectl create -f storageclass.yaml

Create a PVC:

kubectl create -f pvc.yaml

Create a VolumeSnapshotClass:

kubectl create -f snapshotclass.yaml

Create a VolumeSnapshot:

kubectl create -f snapshot.yaml

Create a PVC from a VolumeSnapshot:

kuberctl create -f restore.yaml

If a PersistentVolumeClaim is not bound, the attempt to create a volume snapshot from that PersistentVolumeClaim will fail. No retries will be attempted. An event will be logged to indicate that the PersistentVolumeClaim is not bound.

Note that this could happen if the PersistentVolumeClaim spec and the VolumeSnapshot spec are in the same YAML file. In this case, when the VolumeSnapshot object is created, the PersistentVolumeClaim object is created but volume creation is not complete and therefore the PersistentVolumeClaim is not yet bound. You must wait until the PersistentVolumeClaim is bound and then create the snapshot.

See the Drivers for a list of CSI drivers that implement the snapshot feature.

Traditionally, volumes that are backed by CSI drivers can only be used with a PersistentVolume and PersistentVolumeClaim object combination. This feature supports ephemeral storage use cases and allows CSI volumes to be specified directly in the pod specification. At runtime, nested inline volumes follow the ephemeral lifecycle of their associated pods where the driver handles all phases of volume operations as pods are created and destroyed.

See the design document for futher information.

A pod spec with an ephemeral inline CSI volume. Note that because the volume is expected to be ephemeral, the volumeHandle is not provided. Instead, an ID will be generated by kubelet as volume is mounted for pod use. The generated ID is passed to the CSI driver at NodePublish.

apiVersion: v1
kind: Pod
metadata:
  name: some-pod
spec:
  containers:
    ...
  volumes:
      - name: vol
        csi:
          driver: inline.storage.kubernetes.io
          volumeAttributes:
              foo: bar

Drivers must be modified (or implemented specifically) to support inline ephemeral workflows. When Kubernetes encounters an inline CSI volume embedded in a pod spec, it treats that volume differently. Mainly, the driver will only receive NodePublish, during the volume's mount phase, and NodeUnpublish when the pod is going away and the volume is unmounted. To support inline, a driver must implement the followings:

• Identity service
• Node service

Kubernetes 1.16 only allows using a CSI driver for an inline volume if its CSIDriverInfo object explicitly declares that the driver supports that kind of usage in its volumeLifecycleModes field. This is a safeguard against accidentally using a driver the wrong way.

To use inline volume, Kubernetes 1.15 binaries must start with the CSIInlineVolume feature gate enabled:

--feature-gates=CSIInlineVolume=true

Kubernetes >= 1.16 no longer needs this as the feature is enabled by default.

• CSI Hostpath driver - an example driver that supports both modes and determines the mode on a case-by-case basis (for Kubernetes 1.16) or can be deployed with support for just one of the two modes (for Kubernetes 1.15).
• Image populator plugin - an example CSI driver plugin that uses a container image as a volume.

This page describes to CSI driver developers how to deploy their driver onto a Kubernetes cluster.

A CSI driver is typically deployed in Kubernetes as two components: a controller component and a per-node component.

The controller component can be deployed as a Deployment or StatefulSet on any node in the cluster. It consists of the CSI driver that implements the CSI Controller service and one or more sidecar containers. These controller sidecar containers typically interact with Kubernetes objects and make calls to the driver's CSI Controller service.

It generally does not need direct access to the host and can perform all its operations through the Kubernetes API and external control plane services. Multiple copies of the controller component can be deployed for HA, however it is recommended to use leader election to ensure there is only one active controller at a time.

Controller sidecars include the external-provisioner, external-attacher, external-snapshotter, and external-resizer. Including a sidecar in the deployment may be optional. See each sidecar's page for more details.

Sidecar containers manage Kubernetes events and make the appropriate calls to the CSI driver. The calls are made by sharing a UNIX domain socket through an emptyDir volume between the sidecars and CSI Driver.

Most controller sidecars interact with Kubernetes objects and therefore need to set RBAC policies. Each sidecar repository contains example RBAC configurations.

The node component should be deployed on every node in the cluster through a DaemonSet. It consists of the CSI driver that implements the CSI Node service and the node-driver-registrar sidecar container.

The Kubernetes kubelet runs on every node and is responsible for making the CSI Node service calls. These calls mount and unmount the storage volume from the storage system, making it available to the Pod to consume. Kubelet makes calls to the CSI driver through a UNIX domain socket shared on the host via a HostPath volume. There is also a second UNIX domain socket that the node-driver-registrar uses to register the CSI driver to kubelet.

The node plugin needs direct access to the host for making block devices and/or filesystem mounts available to the Kubernetes kubelet.

The mount point used by the CSI driver must be set to Bidirectional to allow Kubelet on the host to see mounts created by the CSI driver container. See the example below:

      containers:
      - name: my-csi-driver
        ...
        volumeMounts:
        - name: socket-dir
          mountPath: /csi
        - name: mountpoint-dir
          mountPath: /var/lib/kubelet/pods
          mountPropagation: "Bidirectional"
      - name: node-driver-registrar
        ...
        volumeMounts:
        - name: registration-dir
          mountPath: /registration
      volumes:
      # This volume is where the socket for kubelet->driver communication is done
      - name: socket-dir
        hostPath:
          path: /var/lib/kubelet/plugins/<driver-name>
          type: DirectoryOrCreate
      # This volume is where the driver mounts volumes
      - name: mountpoint-dir
        hostPath:
          path: /var/lib/kubelet/pods
          type: Directory
      # This volume is where the node-driver-registrar registers the plugin
      # with kubelet
      - name: registration-dir
        hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory

Deploying a CSI driver onto Kubernetes is highlighted in detail in Recommended Mechanism for Deploying CSI Drivers on Kubernetes.

To use CSI drivers, your Kubernetes cluster must allow privileged pods (i.e. --allow-privileged flag must be set to true for both the API server and the kubelet). This is the default in some environments (e.g. GCE, GKE, kubeadm).

Ensure your API server are started with the privileged flag:

$ ./kube-apiserver ...  --allow-privileged=true ...

$ ./kubelet ...  --allow-privileged=true ...

Note: Starting from Kubernetes 1.13.0, --allow-privileged is true for kubelet. It'll be deprecated in future kubernetes releases.

Another feature that CSI depends on is mount propagation. It allows the sharing of volumes mounted by one container with other containers in the same pod, or even to other pods on the same node. For mount propagation to work, the Docker daemon for the cluster must allow shared mounts. See the [mount propagation docs][mount-propagation-docs] to find out how to enable this feature for your cluster. [This page][docker-shared-mount] explains how to check if shared mounts are enabled and how to configure Docker for shared mounts.

• Simple deployment example using a single pod for all components: see the hostpath example.
• Full deployment example using a DaemonSet for the node plugin and StatefulSet for the controller plugin: TODO

For more information, please read CSI Volume Plugins in Kubernetes Design Doc.

The Hostpath CSI driver is a simple sample driver that provisions a directory on the host. It can be used as an example to get started writing a driver, however it is not meant for production use. The deployment example shows how to deploy and use that driver in Kubernetes.

The example deployment uses the original RBAC rule files that are maintained together with sidecar apps and deploys into the default namespace. A real production should copy the RBAC files and customize them as explained in the comments of those files.

If you encounter any problems, please check the Troubleshooting page.

This section describes how CSI developers can test their CSI drivers.

• Unit Testing
• Functional Testing

The CSI sanity package from csi-test can be used for unit testing your CSI driver.

It contains a set of basic tests that all CSI drivers should pass (for example, NodePublishVolume should fail when no volume id is provided, etc.).

This package can be used in two modes:

• Via a Golang test framework (sanity package is imported as a dependency)
• Via a command line against your driver binary.

Read the documentation of the sanity package for more details.

Drivers should be functionally "end-to-end" tested while deployed in a Kubernetes cluster. Previously, how to do this and what tests to run was left up to driver authors. Now, a standard set of Kubernetes CSI end-to-end tests can be imported and run by third party CSI drivers. This documentation specifies how to do so.

The CSI community is also looking in to establishing an official "CSI Conformance Suite" to recognize "officially certified CSI drivers". This documentation will be updated with more information once that process has been defined.

Currently, csi-sanity exists to help test compliance with the CSI spec, but e2e testing of plugins is needed as well to provide plugin authors and users validation that their plugin is integrated well with specific versions of Kubernetes.

• A Kubernetes v1.13+ Cluster
• Kubectl

There are two ways to run end-to-end tests for your CSI Plugin

1. use Kubernetes E2E Tests, by providing a DriverDefinition YAML file via a parameter.

• Note: In some cases you would not be able to use this method, in running e2e tests by just providing a YAML file defining your CSI plugin. For example the NFS CSI plugin currently does not support dynamic provisoning, so we would want to skip those and run only pre-provisioned tests. For such cases, you would need to write your own testdriver, which is discussed below.

2. import the in-tree storage tests and run them using go test.

This doc will cover how to run the E2E tests using the second method.

In-tree storage e2e tests could be used to test CSI storage plugins. Your repo should be setup similar to how the NFS CSI plugin is setup, where the testfiles are in a test directory and the main test file is in the cmd directory.

To be able to import Kubernetes in-tree storage tests, the CSI plugin would need to use Kubernetes v1.14+ (add to plugin's GoPkg.toml, since pluggable E2E tests become available in v1.14). CSI plugin authors would also be required to implement a testdriver for their CSI plugin. The testdriver provides required functionality that would help setup testcases for a particular plugin.

For any testdriver these functions would be required (Since it implements the TestDriver Interface):

• GetDriverInfo() *testsuites.DriverInfo
• SkipUnsupportedTest(pattern testpatterns.TestPattern)
• PrepareTest(f *framework.Framework) (*testsuites.PerTestConfig, func())

The PrepareTest method is where you would write code to setup your CSI plugin, and it would be called before each test case. It is recommended that you don't deploy your plugin in this method, and rather deploy it manually before running your tests.

GetDriverInfo will return a DriverInfo object that has all of the plugin's capabilities and required information. This object helps tests find the deployed plugin, and also decides which tests should run (depending on the plugin's capabilities).

Here are examples of the NFS and Hostpath DriverInfo objects:

testsuites.DriverInfo{
            Name:        "csi-nfsplugin",
            MaxFileSize: testpatterns.FileSizeLarge,
            SupportedFsType: sets.NewString(
                "", // Default fsType
            ),
            Capabilities: map[testsuites.Capability]bool{
                testsuites.CapPersistence: true,
                testsuites.CapExec:        true,
            },
}

testsuites.DriverInfo{
            Name:        "csi-hostpath",
            FeatureTag:  "",
            MaxFileSize: testpatterns.FileSizeMedium,
            SupportedFsType: sets.NewString(
                "", // Default fsType
            ),
            Capabilities: map[testsuites.Capability]bool{
                testsuites.CapPersistence: true,
            },
}

You would define something similar for your CSI plugin.

SkipUnsupportedTest simply skips any tests that you define there.

Depending on your plugin's specs, you would implement other interaces defined here. For example the NFS testdriver also implements PreprovisionedVolumeTestDriver and PreprovisionedPVTestDriver interfaces, to enable pre-provisoned tests.

After implementing the testdriver for your CSI plugin, you would create a csi-volumes.go file, where the implemented testdriver is used to run in-tree storage testsuites, similar to how the NFS CSI plugin does so. This is where you would define which testsuites you would want to run for your plugin. All available in-tree testsuites can be found here.

Finally, importing the test package into your main test file will initialize the testsuites to run the E2E tests.

The NFS plugin creates a binary to run E2E tests, but you could use go test instead to run E2E tests using a command like this:

go test -v <main test file> -ginkgo.v -ginkgo.progress --kubeconfig=<kubeconfig file> -timeout=0

The following are a set of CSI driver which can be used with Kubernetes:

NOTE: If you would like your driver to be added to this table, please open a pull request in this repo updating this file. Other Features is allowed to be filled in Raw Block, Snapshot, Expansion and Cloning. If driver did not implement any Other Features, please leave it blank.

• [minikube-3378]: Volume mount causes minikube VM to become corrupted

kubectl describe pod your-nodeplugin-pod shows:

failed to start container "your-driver": Error response from daemon:
linux mounts: Path /var/lib/kubelet/pods is mounted on / but it is not a shared mount

Your Docker host is not configured to allow shared mounts. Take a look at this page for instructions to enable them.

If you have a Kubernetes 1.9 cluster, not being able to list VolumeAttachment and the following error are due to the lack of the storage.k8s.io/v1alpha1=true runtime configuration:

$ kubectl logs csi-pod external-attacher
...
I0306 16:34:50.976069       1 reflector.go:240] Listing and watching *v1alpha1.VolumeAttachment from github.com/kubernetes-csi/external-attacher/vendor/k8s.io/client-go/informers/factory.go:86

E0306 16:34:50.992034       1 reflector.go:205] github.com/kubernetes-csi/external-attacher/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1alpha1.VolumeAttachment: the server could not find the requested resource
...

Please see the Kubernetes 1.9 page.

The external components images are under active development. It can happen that they become incompatible with each other. If the issues above above have been ruled out, contact the sig-storage team and/or run the e2e test:

go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=Feature:CSI"