Introduction
Kubernetes Container Storage Interface (CSI) Documentation
This site documents how to develop, deploy, and test a Container Storage Interface (CSI) driver on Kubernetes.
The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code.
The target audience for this site is third-party developers interested in developing CSI drivers for Kubernetes.
Kubernetes users interested in how to deploy or manage an existing CSI driver on Kubernetes should look at the documentation provided by the author of the CSI driver.
Kubernetes users interested in how to use a CSI driver should look at kubernetes.io documentation.
Kubernetes Releases
Kubernetes | CSI Spec Compatibility | Status |
---|---|---|
v1.9 | v0.1.0 | Alpha |
v1.10 | v0.2.0 | Beta |
v1.11 | v0.3.0 | Beta |
v1.13 | v0.3.0, v1.0.0 | GA |
Development and Deployment
Minimum Requirements (for Developing and Deploying a CSI driver for Kubernetes)
Kubernetes is as minimally prescriptive about packaging and deployment of a CSI Volume Driver as possible.
The only requirements are around how Kubernetes (master and node) components find and communicate with a CSI driver.
Specifically, the following is dictated by Kubernetes regarding CSI:
- Kubelet to CSI Driver Communication
- Kubelet directly issues CSI calls (like
NodeStageVolume
,NodePublishVolume
, etc.) to CSI drivers via a Unix Domain Socket to mount and unmount volumes. - Kubelet discovers CSI drivers (and the Unix Domain Socket to use to interact with a CSI driver) via the kubelet plugin registration mechanism.
- Therefore, all CSI drivers deployed on Kubernetes MUST register themselves using the kubelet plugin registration mechanism on each supported node.
- Kubelet directly issues CSI calls (like
- Master to CSI Driver Communication
- Kubernetes master components do not communicate directly (via a Unix Domain Socket or otherwise) with CSI drivers.
- Kubernetes master components interact only with the Kubernetes API.
- Therefore, CSI drivers that require operations that depend on the Kubernetes API (like volume create, volume attach, volume snapshot, etc.) MUST watch the Kubernetes API and trigger the appropriate CSI operations against it.
Because these requirements are minimally prescriptive, CSI driver developers are free to implement and deploy their drivers as they see fit.
That said, to ease development and deployment, the mechanism described below is recommended.
Recommended Mechanism (for Developing and Deploying a CSI driver for Kubernetes)
The Kubernetes development team has established a "Recommended Mechanism" for developing, deploying, and testing CSI Drivers on Kubernetes. It aims to reduce boilerplate code and simplify the overall process for CSI Driver developers.
This "Recommended Mechanism" makes use of the following components:
- Kubernetes CSI Sidecar Containers
- Kubernetes CSI objects
- CSI Driver Testing tools
To implement a CSI driver using this mechanism, a CSI driver developer should:
- Create a containerized application implementing the Identity, Node, and optionally the Controller services described in the CSI specification (the CSI driver container).
- See Developing CSI Driver for more information.
- Unit test it using csi-sanity.
- See Driver - Unit Testing for more information.
- Define Kubernetes API YAML files that deploy the CSI driver container along with appropriate sidecar containers.
- See Deploying in Kubernetes for more information.
- Deploy the driver on a Kubernetes cluster and run end-to-end functional tests on it.
Reference Links
Developing CSI Driver for Kubernetes
The first step to creating a CSI driver is writing an application implementing the gRPC services described in the CSI specification
At a minimum, CSI drivers must implement the following CSI services:
- CSI
Identity
service- Enables callers (Kubernetes components and CSI sidecar containers) to identify the driver and what optional functionality it supports.
- CSI
Node
service- Only
NodePublishVolume
,NodeUnpublishVolume
, andNodeGetCapabilities
are required. - Required methods enable callers to make a volume available at a specified path and discover what optional functionality the driver supports.
- Only
All CSI services may be implemented in the same CSI driver application. The CSI driver application should be containerized to make it easy to deploy on Kubernetes. Once containerized, the CSI driver can be paired with CSI Sidecar Containers and deployed in node and/or controller mode as appropriate.
Capabilities
If your driver supports additional features, CSI "capabilities" can be used to advertise the optional methods/services it supports, for example:
CONTROLLER_SERVICE
(PluginCapability
)- The entire CSI
Controller
service is optional. This capability indicates the driver implement one or more of the methods in the CSIController
service.
- The entire CSI
VOLUME_ACCESSIBILITY_CONSTRAINTS
(PluginCapability
)- This capability indicates the volumes for this driver may not be equally accessible from all nodes in the cluster, and that the driver will return additional topology related information that Kubernetes can use to schedule workloads more intelligently or influence where a volume will be provisioned.
VolumeExpansion
(PluginCapability
)- This capability indicates the driver supports resizing (expanding) volumes after creation.
CREATE_DELETE_VOLUME
(ControllerServiceCapability
)- This capability indicates the driver supports dynamic volume provisioning and deleting.
PUBLISH_UNPUBLISH_VOLUME
(ControllerServiceCapability
)- This capability indicates the driver implements
ControllerPublishVolume
andControllerUnpublishVolume
-- operations that correspond to the Kubernetes volume attach/detach operations. This may, for example, result in a "volume attach" operation against the Google Cloud control plane to attach the specified volume to the specified node for the Google Cloud PD CSI Driver.
- This capability indicates the driver implements
CREATE_DELETE_SNAPSHOT
(ControllerServiceCapability
)- This capability indicates the driver supports provisioning volume snapshots and the ability to provision new volumes using those snapshots.
CLONE_VOLUME
(ControllerServiceCapability
)- This capability indicates the driver supports cloning of volumes.
STAGE_UNSTAGE_VOLUME
(NodeServiceCapability
)- This capability indicates the driver implements
NodeStageVolume
andNodeUnstageVolume
-- operations that correspond to the Kubernetes volume device mount/unmount operations. This may, for example, be used to create a global (per node) volume mount of a block storage device.
- This capability indicates the driver implements
This is an partial list, please see the CSI spec for a complete list of capabilities. Also see the Features section to understand how a feature integrates with Kubernetes.
Kubernetes Changelog
This page summarizes major CSI changes made in each Kubernetes release. For details on individual features, visit the Features section.
Kubernetes 1.17
Breaking Changes
- CSI 0.3 support has been removed. CSI 0.3 drivers will no longer function.
Deprecations
storage.k8s.io/v1beta1
CSINode
object has been deprecated and will be removed in a future release.
Features
- GA
- Volume topology
- Volume limits
- Beta
- Volume snapshots and restore
- CSI migration for AWS EBS and GCE PD drivers
storage.k8s.io/v1
CSINode
object introduced.
Kubernetes 1.16
Features
- Beta
- Volume cloning
- Volume expansion
- Ephemeral local volumes
Kubernetes 1.15
Features
- Volume capacity usage metrics
- Alpha
- Volume cloning
- Ephemeral local volumes
- Resizing secrets
Kubernetes 1.14
Breaking Changes
csi.storage.k8s.io/v1alpha1
CSINodeInfo
andCSIDriver
CRDs are no longer supported.
Features
- Beta
- Topology
- Raw block
- Skip attach
- Pod info on mount
- Alpha
- Volume expansion
storage.k8s.io/v1beta1
CSINode
andCSIDriver
objects introduced.
Kubernetes 1.13
Deprecations
- CSI spec 0.2 and 0.3 are deprecated and support will be removed in Kubernetes 1.17.
Features
- GA support added for CSI spec 1.0.
Kubernetes 1.12
Breaking Changes
Kubelet device plugin registration is enabled by default, which requires CSI
plugins to use driver-registrar:v0.3.0
to register with kubelet.
Features
- Alpha
- Snapshots
- Topology
- Skip attach
- Pod info on mount
csi.storage.k8s.io/v1alpha1
CSINodeInfo
andCSIDriver
CRDs were introduced and have to be installed before deploying a CSI driver.
Kubernetes 1.11
Features
- Beta support added for CSI spec 0.3.
- Alpha
- Raw block
Kubernetes 1.10
Breaking Changes
- CSI spec 0.1 is no longer supported.
Features
- Beta support added for CSI spec 0.2.
This added optional
NodeStageVolume
andNodeUnstageVolume
calls which map to KubernetesMountDevice
andUnmountDevice
operations.
Kubernetes 1.9
Features
- Alpha support added for CSI spec 0.1.
Kubernetes CSI Sidecar Containers
Kubernetes CSI Sidecar Containers are a set of standard containers that aim to simplify the development and deployment of CSI Drivers on Kubernetes.
These containers contain common logic to watch the Kubernetes API, trigger appropriate operations against the “CSI volume driver” container, and update the Kubernetes API as appropriate.
The containers are intended to be bundled with third-party CSI driver containers and deployed together as pods.
The containers are developed and maintained by the Kubernetes Storage community.
Use of the containers is strictly optional, but highly recommended.
Benefits of these sidecar containers include:
- Reduction of "boilerplate" code.
- CSI Driver developers do not have to worry about complicated, "Kubernetes specific" code.
- Separation of concerns.
- Code that interacts with the Kubernetes API is isolated from (and in a different container then) the code that implements the CSI interface.
The Kubernetes development team maintains the following Kubernetes CSI Sidecar Containers:
- external-provisioner
- external-attacher
- external-snapshotter
- external-resizer
- node-driver-registrar
- cluster-driver-registrar (deprecated)
- livenessprobe
Kubernetes and CSI Sidecar Compatibility
Every version of a sidecar has a minimum, maximum and recommended Kubernetes version that it is compatible with.
Minimum Version
Minimum version specifies the lowest Kubernetes version where the sidecar will function with the most basic functionality, and no additional features added later. Generally, this aligns with the Kubernetes version where that CSI spec version was added.
Maximum Version
Similarly, the max Kubernetes version generally aligns with when support for that CSI spec version was removed or if a particular Kubernetes API or feature was deprecated and removed.
Recommended Version
It is important to note that any new features added to the sidecars may have dependencies on Kubernetes versions greater than the minimum Kubernetes version. The recommended Kubernetes version specifies the lowest Kubernetes version needed where all features of a sidecar will function correctly. Trying to use a new sidecar feature on a Kubernetes cluster below the recommended Kubernetes version may fail to function correctly. For that reason, it is encouraged to stay as close to the recommended Kubernetes version as possible.
For more details on which features are supported with which Kubernetes versions and their corresponding sidecars, please see each feature's individual page.
Alpha Features
It is also important to note that alpha features are subject to break or be removed across Kubernetes and sidecar releases. There is no guarantee alpha features will continue to function if upgrading the Kubernetes cluster or upgrading a sidecar.
CSI external-attacher
Status and Releases
Git Repository: https://github.com/kubernetes-csi/external-attacher
Status: GA/Stable
Latest stable release | Branch | Min CSI Version | Max CSI Version | Container Image | Min K8s Version | Max K8s Version | Recommended K8s Version |
---|---|---|---|---|---|---|---|
external-attacher v2.0.0 | release-2.0 | v1.0.0 | - | quay.io/k8scsi/csi-attacher:v2.0.0 | v1.14 | - | v1.15 |
external-attacher v1.2.1 | release-1.2 | v1.0.0 | - | quay.io/k8scsi/csi-attacher:v1.2.1 | v1.13 | - | v1.15 |
external-attacher v1.1.1 | release-1.1 | v1.0.0 | - | quay.io/k8scsi/csi-attacher:v1.1.1 | v1.13 | - | v1.14 |
external-attacher v0.4.2 | release-0.4 | v0.3.0 | v0.3.0 | quay.io/k8scsi/csi-attacher:v0.4.2 | v1.10 | v1.16 | v1.10 |
Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page
Description
The CSI external-attacher
is a sidecar container that watches the Kubernetes API server for VolumeAttachment
objects and triggers Controller[Publish|Unpublish]Volume
operations against a CSI endpoint.
Usage
CSI drivers that require integrating with the Kubernetes volume attach/detach hooks should use this sidecar container, and advertise the CSI PUBLISH_UNPUBLISH_VOLUME
controller capability.
For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-attacher/blob/master/README.md.
Deployment
The CSI external-attacher
is deployed as a controller. See deployment section for more details.
CSI external-provisioner
Status and Releases
Git Repository: https://github.com/kubernetes-csi/external-provisioner
Status: GA/Stable
Latest stable release | Branch | Min CSI Version | Max CSI Version | Container Image | Min K8s Version | Max K8s Version | Recommended K8s Version |
---|---|---|---|---|---|---|---|
external-provisioner v1.4.0 | release-1.4 | v1.0.0 | - | quay.io/k8scsi/csi-provisioner:v1.4.0 | v1.13 | - | v1.16 |
external-provisioner v1.3.1 | release-1.3 | v1.0.0 | - | quay.io/k8scsi/csi-provisioner:v1.3.1 | v1.13 | v1.19 | v1.15 |
external-provisioner v1.2.0 | release-1.2 | v1.0.0 | - | quay.io/k8scsi/csi-provisioner:v1.2.0 | v1.13 | v1.19 | v1.14 |
external-provisioner v0.4.2 | release-0.4 | v0.3.0 | v0.3.0 | quay.io/k8scsi/csi-provisioner:v0.4.2 | v1.10 | v1.16 | v1.10 |
Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page
Description
The CSI external-provisioner
is a sidecar container that watches the Kubernetes API server for PersistentVolumeClaim
objects.
It calls CreateVolume
against the specified CSI endpoint to provision a new volume.
Volume provisioning is triggered by the creation of a new Kubernetes PersistentVolumeClaim
object, if the PVC references a Kubernetes StorageClass
, and the name in the provisioner
field of the storage class matches the name returned by the specified CSI endpoint in the GetPluginInfo
call.
Once a new volume is successfully provisioned, the sidecar container creates a Kubernetes PersistentVolume
object to represent the volume.
The deletion of a PersistentVolumeClaim
object bound to a PersistentVolume
corresponding to this driver with a delete
reclaim policy causes the sidecar container to trigger a DeleteVolume
operation against the specified CSI endpoint to delete the volume. Once the volume is successfully deleted, the sidecar container also deletes the PersistentVolume
object representing the volume.
DataSources
The external-provisioner provides the ability to request a volume be pre-populated from a data source during provisioning. For more information on how data sources are handled see DataSources.
Snapshot
The CSI external-provisioner
supports the Snapshot
DataSource. If a Snapshot
CRD is specified as a data source on a PVC object, the sidecar container fetches the information about the snapshot by fetching the SnapshotContent
object and populates the data source field in the resulting CreateVolume
call to indicate to the storage system that the new volume should be populated using the specified snapshot.
PersistentVolumeClaim (clone)
Cloning is also implemented by specifying a kind:
of type PersistentVolumeClaim
in the DataSource field of a Provision request. It's the responsbility of the external-provisioner to verify that the claim specified in the DataSource object exists, is in the same storage class as the volume being provisioned and that the claim is currently Bound
.
StorageClass Parameters
When provisioning a new volume, the CSI external-provisioner
sets the map<string, string> parameters
field in the CSI CreateVolumeRequest
call to the key/values specified in the StorageClass
it is handling.
The CSI external-provisioner
(v1.0.1+) also reserves the parameter keys prefixed with csi.storage.k8s.io/
. Any keys prefixed with csi.storage.k8s.io/
are not passed to the CSI driver as an opaque parameter
.
The following reserved StorageClass
parameter keys trigger behavior in the CSI external-provisioner
:
csi.storage.k8s.io/provisioner-secret-name
csi.storage.k8s.io/provisioner-secret-namespace
csi.storage.k8s.io/controller-publish-secret-name
csi.storage.k8s.io/controller-publish-secret-namespace
csi.storage.k8s.io/node-stage-secret-name
csi.storage.k8s.io/node-stage-secret-namespace
csi.storage.k8s.io/node-publish-secret-name
csi.storage.k8s.io/node-publish-secret-namespace
csi.storage.k8s.io/fstype
If the PVC VolumeMode
is set to Filesystem
, and the value of csi.storage.k8s.io/fstype
is specified, it is used to populate the FsType
in CreateVolumeRequest.VolumeCapabilities[x].AccessType
and the AccessType
is set to Mount
.
For more information on how secrets are handled see Secrets & Credentials.
Example StorageClass
:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gold-example-storage
provisioner: exampledriver.example.com
parameters:
disk-type: ssd
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/provisioner-secret-name: mysecret
csi.storage.k8s.io/provisioner-secret-namespace: mynamespace
Usage
CSI drivers that support dynamic volume provisioning should use this sidecar container, and advertise the CSI CREATE_DELETE_VOLUME
controller capability.
For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-provisioner/blob/master/README.md.
Deployment
The CSI external-provisioner
is deployed as a controller. See deployment section for more details.
CSI external-resizer
Status and Releases
Git Repository: https://github.com/kubernetes-csi/external-resizer
Status: Beta starting with v0.3.0
Latest release | Branch | Min CSI Version | Max CSI Version | Container Image | Min K8s Version | Max K8s Version | Recommended K8s Version |
---|---|---|---|---|---|---|---|
external-resizer v0.3.0 | master | v1.1.0 | - | quay.io/k8scsi/csi-resizer:v0.3.0 | v1.15 | - | v1.16 |
external-resizer v0.2.0 | master | v1.1.0 | - | quay.io/k8scsi/csi-resizer:v0.2.0 | v1.15 | - | v1.15 |
external-resizer v0.1.0 | master | v1.1.0 | - | quay.io/k8scsi/csi-resizer:v0.1.0 | v1.14 | v1.14 | v1.14 |
Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page
Description
The CSI external-resizer
is a sidecar container that watches the Kubernetes API server for PersistentVolumeClaim
object edits and
triggers ControllerExpandVolume
operations against a CSI endpoint if user requested more storage on PersistentVolumeClaim
object.
Usage
CSI drivers that support Kubernetes volume expansion should use this sidecar container, and advertise the CSI VolumeExpansion
plugin capability.
Deployment
The CSI external-resizer
is deployed as a controller. See deployment section for more details.
CSI external-snapshotter
Status and Releases
Git Repository: https://github.com/kubernetes-csi/external-snapshotter
Status: Alpha
Latest stable release | Branch | Min CSI Version | Max CSI Version | Container Image | Min K8s Version | Max K8s Version | Recommended K8s Version |
---|---|---|---|---|---|---|---|
external-snapshotter v1.2.2 | release-1.2 | v1.0.0 | - | quay.io/k8scsi/csi-snapshotter:v1.2.2 | v1.13 | - | v1.14 |
external-snapshotter v0.4.1 | release-0.4 | v0.3.0 | v0.3.0 | quay.io/k8scsi/csi-snapshotter:v0.4.1 | v1.10 | -v1.16 | v1.10 |
Definitions of the min/max/recommended Kubernetes versions can be found on the sidecar page
Description
The CSI external-snapshotter
is a sidecar container that watches the Kubernetes API server for VolumeSnapshot
and VolumeSnapshotContent
CRD objects.
The creation of a new VolumeSnapshot
object referencing a SnapshotClass
CRD object corresponding to this driver causes the sidecar container to trigger a CreateSnapshot
operation against the specified CSI endpoint to provision a new snapshot. When a new snapshot is successfully provisioned, the sidecar container creates a Kubernetes VolumeSnapshotContent
object to represent the new snapshot.
The deletion of a VolumeSnapshot
object bound to a VolumeSnapshotContent
corresponding to this driver with a delete
reclaim policy causes the sidecar container to trigger a DeleteSnapshot
operation against the specified CSI endpoint to delete the snapshot. Once the snapshot is successfully deleted, the sidecar container also deletes the VolumeSnapshotContent
object representing the snapshot.
For detailed information about volume snapshot and restore functionality, see Volume Snapshot & Restore.
Usage
CSI drivers that support provisioning volume snapshots and the ability to provision new volumes using those snapshots should use this sidecar container, and advertise the CSI CREATE_DELETE_SNAPSHOT
controller capability.
For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md.
Deployment
The CSI external-snapshotter
is deployed as a controller. See deployment section for more details.
For an example deployment, see this example which deploys external-snapshotter
and external-provisioner
with the Hostpath CSI driver.
CSI livenessprobe
Status and Releases
Git Repository: https://github.com/kubernetes-csi/livenessprobe
Status: GA/Stable
Latest stable release | Branch | Compatible with CSI Version | Container Image | Min k8s Version | Max k8s version |
---|---|---|---|---|---|
livenessprobe v1.1.0 | release-1.1 | v1.0.0 | quay.io/k8scsi/livenessprobe:v1.1.0 | v1.13 | - |
Unsupported. | No 0.x branch. | v0.3.0 | quay.io/k8scsi/livenessprobe:v0.4.1 | v1.10 | v1.16 |
Description
The CSI livenessprobe
is a sidecar container that monitors the health of the CSI driver and reports it to Kubernetes via the Liveness Probe mechanism. This enables Kubernetes to automatically detect issues with the driver and restart the pod to try and fix the issue.
Usage
All CSI drivers should use the liveness probe to improve the availability of the driver while deployed on Kubernetes.
For detailed information (binary parameters, RBAC rules, etc.), see https://github.com/kubernetes-csi/livenessprobe/blob/master/README.md.
Deployment
The CSI livenessprobe
is deployed as part of controller and node deployments. See deployment section for more details.
CSI node-driver-registrar
Status and Releases
Git Repository: https://github.com/kubernetes-csi/node-driver-registrar
Status: GA/Stable
Latest stable release | Branch | Min CSI Version | Max CSI Version | Container Image | Min k8s Version | Max k8s version |
---|---|---|---|---|---|---|
node-driver-registrar v1.2.0 | release-1.2 | v1.0.0 | - | quay.io/k8scsi/csi-node-driver-registrar:v1.2.0 | v1.13 | - |
driver-registrar v0.4.2 | release-0.4 | v0.3.0 | v0.3.0 | quay.io/k8scsi/driver-registrar:v0.4.2 | v1.10 | v1.16 |
Description
The CSI node-driver-registrar
is a sidecar container that fetches driver information (using NodeGetInfo
) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.
Usage
Kubelet directly issues CSI NodeGetInfo
, NodeStageVolume
, and NodePublishVolume
calls against CSI drivers. It uses the kubelet plugin registration mechanism to discover the unix domain socket to talk to the CSI driver. Therefore, all CSI drivers should use this sidecar container to register themselves with kubelet.
For detailed information (binary parameters, etc.), see the README of the relevant branch.
Deployment
The CSI node-driver-registrar
is deployed per node. See deployment section for more details.
CSI cluster-driver-registrar
Deprecated
This sidecar container was not updated since Kubernetes 1.13. As of Kubernetes 1.16, this side car container is officially deprecated.
The purpose of this side car container was to automatically register a CSIDriver object containing information about the driver with Kubernetes. Without this side car, developers and CSI driver vendors will now have to add a CSIDriver object in their installation manifest or any tool that installs their CSI driver.
Please see CSIDriver for more information.
Status and Releases
Git Repository: https://github.com/kubernetes-csi/cluster-driver-registrar
Status: Alpha
Latests stable release | Branch | Compatible with CSI Version | Container Image | Min k8s Version | Max k8s version |
---|---|---|---|---|---|
cluster-driver-registrar v1.0.1 | release-1.0 | v1.0.0 | quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1 | v1.13 | - |
driver-registrar v0.4.2 | release-0.4 | v0.3.0 | quay.io/k8scsi/driver-registrar:v0.4.2 | v1.10 | - |
Description
The CSI cluster-driver-registrar
is a sidecar container that registers a CSI Driver with a Kubernetes cluster by creating a CSIDriver Object which enables the driver to customize how Kubernetes interacts with it.
Usage
CSI drivers that use one of the following Kubernetes features should use this sidecar container:
- Skip Attach
- For drivers that don't support
ControllerPublishVolume
, this indicates to Kubernetes to skip the attach operation and eliminates the need to deploy theexternal-attacher
sidecar.
- For drivers that don't support
- Pod Info on Mount
- This causes Kubernetes to pass metadata such as Pod name and namespace to the
NodePublishVolume
call.
- This causes Kubernetes to pass metadata such as Pod name and namespace to the
If you are not using one of these features, this sidecar container (and the creation of the CSIDriver Object) is not required. However, it is still recommended, because the CSIDriver Object makes it easier for users to easily discover the CSI drivers installed on their clusters.
For detailed information (binary parameters, etc.), see the README of the relevant branch.
Deployment
The CSI cluster-driver-registrar
is deployed as a controller. See deployment section for more details.
CSI objects
Status: Beta
The Kubernetes API contains the following CSI specific objects:
Both are part of storage.k8s.io/v1beta1
API group.
The schema definition for the objects can be found here: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/storage/types.go
CSIDriver Object
Status
- Kubernetes 1.12 - 1.13: Alpha
- Kubernetes 1.14: Beta
What is the CSIDriver object?
The CSIDriver
Kubernetes API object serves two purposes:
- Simplify driver discovery
- If a CSI driver creates a
CSIDriver
object, Kubernetes users can easily discover the CSI Drivers installed on their cluster (simply by issuingkubectl get CSIDriver
)
- Customizing Kubernetes behavior
- Kubernetes has a default set of behaviors when dealing with CSI Drivers (for example, it calls the
Attach
/Detach
operations by default). This object allows CSI drivers to specify how Kubernetes should interact with it.
What fields does the CSIDriver
object have?
Here is an example of a v1beta1 CSIDriver
object:
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: mycsidriver.example.com
spec:
attachRequired: true
podInfoOnMount: true
volumeLifecycleModes: # added in Kubernetes 1.16
- Persistent
- Ephemeral
There are three important fields:
name
- This should correspond to the full name of the CSI driver.
attachRequired
- Indicates this CSI volume driver requires an attach operation (because it implements the CSI
ControllerPublishVolume
method), and that Kubernetes should call attach and wait for any attach operation to complete before proceeding to mounting. - If a
CSIDriver
object does not exist for a given CSI Driver, the default istrue
-- meaning attach will be called. - If a
CSIDriver
object exists for a given CSI Driver, but this field is not specified, it also defaults totrue
-- meaning attach will be called. - For more information see Skip Attach.
- Indicates this CSI volume driver requires an attach operation (because it implements the CSI
podInfoOnMount
- Indicates this CSI volume driver requires additional pod information (like pod name, pod UID, etc.) during mount operations.
- If value is not specified or
false
, pod information will not be passed on mount. - If value is set to
true
, Kubelet will pass pod information asvolume_context
in CSINodePublishVolume
calls:"csi.storage.k8s.io/pod.name": pod.Name
"csi.storage.k8s.io/pod.namespace": pod.Namespace
"csi.storage.k8s.io/pod.uid": string(pod.UID)
- For more information see Pod Info on Mount.
volumeLifecycleModes
- This field was added in Kubernetes 1.16 and cannot be set when using an older Kubernetes release.
- It informs Kubernetes about the volume modes that are supported by the driver.
This ensures that the driver is not used incorrectly by users.
The default is
Persistent
, which is the normal PVC/PV mechanism.Ephemeral
enables inline ephemeral volumes in addition (when both are listed) or instead of normal volumes (when it is the only entry in the list).
What creates the CSIDriver object?
To install, a CSI driver's deployment manifest must contain a CSIDriver
object as shown in the example above.
NOTE: The cluster-driver-registrar side-car which was used to create CSIDriver objects in Kubernetes 1.13 has been deprecated for Kubernetes 1.16. No cluster-driver-registrar has been released for Kubernetes 1.14 and later.
CSIDriver
instance should exist for whole lifetime of all pods that use
volumes provided by corresponding CSI driver, so Skip Attach
and Pod Info on Mount features work correctly.
Listing registered CSI drivers
Using the CSIDriver
object, it is now possible to query Kubernetes to get a list of registered drivers running in the cluster as shown below:
$> kubectl get csidrivers.storage.k8s.io
NAME CREATED AT
hostpath.csi.k8s.io 2019-09-13T09:58:43Z
Or get a more detailed view of your registered driver with:
$> kubectl describe csidrivers.storage.k8s.io
Name: hostpath.csi.k8s.io
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"storage.k8s.io/v1beta1","kind":"CSIDriver","metadata":{"annotations":{},"name":"hostpath.csi.k8s.io"},"spec":{"podInfoOnMou...
API Version: storage.k8s.io/v1beta1
Kind: CSIDriver
Metadata:
Creation Timestamp: 2019-09-13T09:58:43Z
Resource Version: 341
Self Link: /apis/storage.k8s.io/v1beta1/csidrivers/hostpath.csi.k8s.io
UID: 1860f2a1-85f8-4357-a933-c45e54f0c8e0
Spec:
Attach Required: true
Pod Info On Mount: true
Volume Lifecycle Modes:
Persistent
Ephemeral
Events: <none>
Changes from Alpha to Beta
CRD to Built in Type
During alpha development, the CSIDriver
object was also defined as a Custom Resource Definition (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
In the move from alpha to beta, the API Group for this object changed from csi.storage.k8s.io/v1alpha1
to storage.k8s.io/v1beta1
.
There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.
Enabling CSIDriver on Kubernetes
In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of CSIDriver
on these versions, do the following:
- Ensure the feature gate is enabled via the following Kubernetes feature flag:
--feature-gates=CSIDriverRegistry=true
- Either ensure the
CSIDriver
CRD is automatically installed via the Kubernetes Storage CRD addon OR manually install theCSIDriver
CRD on the Kubernetes cluster with the following command:
$> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csidriver.yaml
Kubernetes v1.14+, uses the same Kubernetes feature flag, but because the feature is beta, it is enabled by default. And since the API type (as of beta) is built in to the Kubernetes API, installation of the CRD is no longer required.
CSINode Object
Status
- Kubernetes 1.12 - 1.13: Alpha
- Kubernetes 1.14: Beta
What is the CSINode object?
CSI drivers generate node specific information. Instead of storing this in the Kubernetes Node
API Object, a new CSI specific Kubernetes CSINode
object was created.
It serves the following purposes:
- Mapping Kubernetes node name to CSI Node name,
- The CSI
GetNodeInfo
call returns the name by which the storage system refers to a node. Kubernetes must use this name in futureControllerPublishVolume
calls. Therefore, when a new CSI driver is registered, Kubernetes stores the storage system node ID in theCSINode
object for future reference.
- Driver availability
- A way for kubelet to communicate to the kube-controller-manager and kubernetes scheduler whether the driver is available (registered) on the node or not.
- Volume topology
- The CSI
GetNodeInfo
call returns a set of keys/values labels identifying the topology of that node. Kubernetes uses this information to do topology-aware provisioning (see PVC Volume Binding Modes for more details). It stores the key/values as labels on the Kubernetes node object. In order to recall whichNode
label keys belong to a specific CSI driver, the kubelet stores the keys in theCSINode
object for future reference.
What fields does the CSINode object have?
Here is an example of a v1beta1 CSINode
object:
apiVersion: storage.k8s.io/v1beta1
kind: CSINodeInfo
metadata:
name: node1
spec:
drivers:
- name: mycsidriver.example.com
nodeID: storageNodeID1
topologyKeys: ['mycsidriver.example.com/regions', "mycsidriver.example.com/zones"]
What the fields mean:
drivers
- list of CSI drivers running on the node and their properties.name
- the CSI driver that this object refers to.nodeID
- the assigned identifier for the node as determined by the driver.topologyKeys
- A list of topology keys assigned to the node as supported by the driver.
What creates the CSINode object?
CSI drivers do not need to create the CSINode
object directly. Instead they should use the node-driver-registrar sidecar container. This sidecar container will interact with kubelet via the kubelet plugin registration mechanism to automatically populate the CSINode
object on behalf of the the CSI driver.
Changes from Alpha to Beta
CRD to Built in Type
The alpha object was called CSINodeInfo
, whereas the beta object is called
CSINode
. The alpha CSINodeInfo
object was also defined as a Custom Resource Definition (CRD). As part of the promotion to beta the object has been moved to the built-in Kubernetes API.
In the move from alpha to beta, the API Group for this object changed from csi.storage.k8s.io/v1alpha1
to storage.k8s.io/v1beta1
.
There is no automatic update of existing CRDs and their CRs during Kubernetes update to the new build-in type.
Enabling CSINodeInfo on Kubernetes
In Kubernetes v1.12 and v1.13, because the feature was alpha, it was disabled by default. To enable the use of CSINodeInfo
on these versions, do the following:
- Ensure the feature gate is enabled with
--feature-gates=CSINodeInfo=true
- Either ensure the
CSIDriver
CRD is automatically installed via the Kubernetes Storage CRD addon OR manually install theCSINodeInfo
CRD on the Kubernetes cluster with the following command:
$> kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/master/pkg/crd/manifests/csinodeinfo.yaml
Kubernetes v1.14+, uses the same Kubernetes feature flag, but because the feature is beta, it is enabled by default. And since the API type (as of beta) is built in to the Kubernetes API, installation of the CRD is no longer required.
Features
The Kubernetes implementation of CSI has multiple sub-features. This section describes these sub-features, their status (although support for CSI in Kubernetes is GA/stable, support of sub-features moves independently so sub-features maybe alpha or beta), and how to integrate them in to your CSI Driver.
Secrets and Credentials
Some drivers may require a secret in order to complete operations.
CSI Driver Secrets
If a CSI Driver requires secrets for a backend (a service account, for example), and this secret is required at the "per driver" granularity (not different "per CSI operation" or "per volume"), then the secret SHOULD be injected directly in to CSI driver pods via standard Kubernetes secret distribution mechanisms during deployment.
CSI Operation Secrets
If a CSI Driver requires secrets "per CSI operation" or "per volume" or "per storage pool", the CSI spec allows secrets to be passed in for various CSI operations (including CreateVolumeRequest
, ControllerPublishVolumeRequest
, and more).
Cluster admins can populate such secrets by creating Kubernetes Secret
objects and specifying the keys in the StorageClass
or SnapshotClass
objects.
The CSI sidecar containers facilitate the handling of secrets between Kubernetes and the CSI Driver. For more details see:
Secret RBAC Rules
For reducing RBAC permissions as much as possible, secret rules are disabled in each sidecar repository by default.
Please add or update RBAC rules if secret is expected to use.
To set proper secret permission, uncomment related lines defined in rbac.yaml
(e.g. external-provisioner/deploy/kubernetes/rbac.yaml)
Handling Sensitive Information
CSI Drivers that accept secrets SHOULD handle this data carefully. It may contain sensitive information and MUST be treated as such (e.g. not logged).
To make it easier to handle secret fields (e.g. strip them from CSI protos when logging), the CSI spec defines a decorator (csi_secret
) on all fields containing sensitive information. Any fields decorated with csi_secret
MUST be treated as if they contain sensitive information (e.g. not logged, etc.).
The Kubernetes CSI development team also provides a GO lang package called protosanitizer
that CSI driver developers may be used to remove values for all fields in a gRPC messages decorated with csi_secret
. The library can be found in kubernetes-csi/csi-lib-utils/protosanitizer. The Kubernetes CSI Sidecar Containers and sample drivers use this library to ensure no sensitive information is logged.
StorageClass Secrets
The CSI external-provisioner sidecar container facilitates the handling of secrets for the following operations:
CreateVolumeRequest
DeleteVolumeRequest
ControllerPublishVolumeRequest
ControllerUnpublishVolumeRequest
ControllerExpandVolumeRequest
NodeStageVolumeRequest
NodePublishVolumeRequest
CSI external-provisioner
v1.0.1+ supports the following keys in StorageClass.parameters
:
csi.storage.k8s.io/provisioner-secret-name
csi.storage.k8s.io/provisioner-secret-namespace
csi.storage.k8s.io/controller-publish-secret-name
csi.storage.k8s.io/controller-publish-secret-namespace
csi.storage.k8s.io/node-stage-secret-name
csi.storage.k8s.io/node-stage-secret-namespace
csi.storage.k8s.io/node-publish-secret-name
csi.storage.k8s.io/node-publish-secret-namespace
CSI external-provisioner
v1.2.0+ adds support for the following keys in StorageClass.parameters
:
csi.storage.k8s.io/controller-expand-secret-name
csi.storage.k8s.io/controller-expand-secret-namespace
Cluster admins can populate the secret fields for the operations listed above with data from Kubernetes Secret
objects by specifying these keys in the StorageClass
object.
Examples
Basic Provisioning Secret
In this example, the external-provisioner will fetch Kubernetes Secret
object fast-storage-provision-key
in the namespace pd-ssd-credentials
and pass the credentials to the CSI driver named csi-driver.team.example.com
in the CreateVolume
CSI call.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast-storage
provisioner: csi-driver.team.example.com
parameters:
type: pd-ssd
csi.storage.k8s.io/provisioner-secret-name: fast-storage-provision-key
csi.storage.k8s.io/provisioner-secret-namespace: pd-ssd-credentials
All volumes provisioned using this StorageClass
use the same secret.
Per Volume Secrets
In this example, the external-provisioner will generate the name of the Kubernetes Secret
object and namespace for the NodePublishVolume
CSI call, based on the PVC namespace and annotations, at volume provision time.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast-storage
provisioner: csi-driver.team.example.com
parameters:
type: pd-ssd
csi.storage.k8s.io/node-publish-secret-name: ${pvc.annotations['team.example.com/key']}
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
This StorageClass will result in the creation of a PersistentVolume
API object referencing a "node publish secret" in the same namespace as the PersistentVolumeClaim
that triggered the provisioning and with a name specified as an annotation on the PersistentVolumeClaim
. This could be used to give the creator of the PersistentVolumeClaim
the ability to specify a secret containing a decryption key they have control over.
Multiple Operation Secrets
A drivers may support secret keys for multiple operations. In this case, you can provide secrets references for each operation:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast-storage-all
provisioner: csi-driver.team.example.com
parameters:
type: pd-ssd
csi.storage.k8s.io/provisioner-secret-name: ${pvc.name}
csi.storage.k8s.io/provisioner-secret-namespace: ${pvc.namespace}-fast-storage
csi.storage.k8s.io/node-publish-secret-name: ${pvc.name}-${pvc.annotations['team.example.com/key']}
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}-fast-storage
Operations
Details for each secret supported by the external-provisioner can be found below.
Create/Delete Volume Secret
The CSI external-provisioner
(v1.0.1+) looks for the following keys in StorageClass.parameters
.
csi.storage.k8s.io/provisioner-secret-name
csi.storage.k8s.io/provisioner-secret-namespace
The values of both of these parameters, together, refer to the name and namespace of a Secret
object in the Kubernetes API.
If specified, the CSI external-provisioner
will attempt to fetch the secret before provisioning and deletion.
If the secret is retrieved successfully, the provisioner passes it to the CSI driver in the CreateVolumeRequest.secrets
or DeleteVolumeRequest.secrets
field.
If no such secret exists in the Kubernetes API, or the provisioner is unable to fetch it, the provision operation will fail.
Note, however, that the delete operation will continue even if the secret is not found (because, for example, the entire namespace containing the secret was deleted). In this case, if the driver requires a secret for deletion, then the volume and PV may need to be manually cleaned up.
The values of these parameters may be "templates". The external-provisioner
will automatically resolve templates at volume provision time, as detailed below:
csi.storage.k8s.io/provisioner-secret-name
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning. - Support added in CSI
external-provisioner
v1.2.0+
- Replaced with namespace of the
${pvc.name}
- Replaced with the name of the
PersistentVolumeClaim
object that triggered provisioning. - Support added in CSI
external-provisioner
v1.2.0+
- Replaced with the name of the
csi.storage.k8s.io/provisioner-secret-namespace
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
Controller Publish/Unpublish Secret
The CSI external-provisioner
(v1.0.1+) looks for the following keys in StorageClass.parameters
:
csi.storage.k8s.io/controller-publish-secret-name
csi.storage.k8s.io/controller-publish-secret-namespace
The values of both of these parameters, together, refer to the name and namespace of a Secret
object in the Kubernetes API.
If specified, the CSI external-provisioner
sets the CSIPersistentVolumeSource.ControllerPublishSecretRef
field in the new PersistentVolume
object to refer to this secret once provisioning is successful.
The CSI external-attacher
then attempts to fetch the secret referenced by the CSIPersistentVolumeSource.ControllerPublishSecretRef
, if specified, before an attach or detach operation.
If no such secret exists in the Kubernetes API, or the external-attacher
is unable to fetch it, the attach or detach operation fails.
If the secret is retrieved successfully, the external-attacher
passes it to the CSI driver in the ControllerPublishVolumeRequest.secrets
or ControllerUnpublishVolumeRequest.secrets
field.
The values of these parameters may be "templates". The external-provisioner
will automatically resolve templates at volume provision time, as detailed below:
csi.storage.k8s.io/controller-publish-secret-name
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
${pvc.name}
- Replaced with the name of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with the name of the
${pvc.annotations['<ANNOTATION_KEY>']}
(e.g.${pvc.annotations['example.com/key']}
)- Replaced with the value of the specified annotation from the
PersistentVolumeClaim
object that triggered provisioning
- Replaced with the value of the specified annotation from the
csi.storage.k8s.io/controller-publish-secret-namespace
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
Node Stage Secret
The CSI external-provisioner
(v1.0.1+) looks for the following keys in StorageClass.parameters
:
csi.storage.k8s.io/node-stage-secret-name
csi.storage.k8s.io/node-stage-secret-namespace
The value of both parameters, together, refer to the name and namespace of the Secret
object in the Kubernetes API.
If specified, the CSI external-provisioner
sets the CSIPersistentVolumeSource.NodeStageSecretRef
field in the new PersistentVolume
object to refer to this secret once provisioning is successful.
The Kubernetes kubelet then attempts to fetch the secret referenced by the CSIPersistentVolumeSource.NodeStageSecretRef
field, if specified, before a mount device operation.
If no such secret exists in the Kubernetes API, or the kubelet is unable to fetch it, the mount device operation fails.
If the secret is retrieved successfully, the kubelet passes it to the CSI driver in the NodeStageVolumeRequest.secrets
field.
The values of these parameters may be "templates". The external-provisioner
will automatically resolve templates at volume provision time, as detailed below:
csi.storage.k8s.io/node-stage-secret-name
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
${pvc.name}
- Replaced with the name of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with the name of the
${pvc.annotations['<ANNOTATION_KEY>']}
(e.g.${pvc.annotations['example.com/key']}
)- Replaced with the value of the specified annotation from the
PersistentVolumeClaim
object that triggered provisioning
- Replaced with the value of the specified annotation from the
csi.storage.k8s.io/node-stage-secret-namespace
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
Node Publish Secret
The CSI external-provisioner
(v1.0.1+) looks for the following keys in StorageClass.parameters
:
csi.storage.k8s.io/node-publish-secret-name
csi.storage.k8s.io/node-publish-secret-namespace
The value of both parameters, together, refer to the name and namespace of the Secret
object in the Kubernetes API.
If specified, the CSI external-provisioner
sets the CSIPersistentVolumeSource.NodePublishSecretRef
field in the new PersistentVolume
object to refer to this secret once provisioning is successful.
The Kubernetes kubelet, attempts to fetch the secret referenced by the CSIPersistentVolumeSource.NodePublishSecretRef
field, if specified, before a mount operation.
If no such secret exists in the Kubernetes API, or the kubelet is unable to fetch it, the mount operation fails.
If the secret is retrieved successfully, the kubelet passes it to the CSI driver in the NodePublishVolumeRequest.secrets
field.
The values of these parameters may be "templates". The external-provisioner
will automatically resolve templates at volume provision time, as detailed below:
csi.storage.k8s.io/node-publish-secret-name
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
${pvc.name}
- Replaced with the name of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with the name of the
${pvc.annotations['<ANNOTATION_KEY>']}
(e.g.${pvc.annotations['example.com/key']}
)- Replaced with the value of the specified annotation from the
PersistentVolumeClaim
object that triggered provisioning
- Replaced with the value of the specified annotation from the
csi.storage.k8s.io/node-publish-secret-namespace
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
Controller Expand (Volume Resize) Secret
The CSI external-provisioner
(v1.2.0+) looks for the following keys in StorageClass.parameters
:
csi.storage.k8s.io/controller-expand-secret-name
csi.storage.k8s.io/controller-expand-secret-namespace
The value of both parameters, together, refer to the name and namespace of the Secret
object in the Kubernetes API.
If specified, the CSI external-provisioner
sets the CSIPersistentVolumeSource.ControllerExpandSecretRef
field in the new PersistentVolume
object to refer to this secret once provisioning is successful.
The external-resizer
(v0.2.0+), attempts to fetch the secret referenced by the CSIPersistentVolumeSource.ControllerExpandSecretRef
field, if specified, before starting a volume resize (expand) operation.
If no such secret exists in the Kubernetes API, or the external-resizer
is unable to fetch it, the resize (expand) operation fails.
If the secret is retrieved successfully, the external-resizer
passes it to the CSI driver in the ControllerExpandVolumeRequest.secrets
field.
The values of these parameters may be "templates". The external-provisioner
will automatically resolve templates at volume provision time, as detailed below:
csi.storage.k8s.io/controller-expand-secret-name
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
${pvc.name}
- Replaced with the name of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with the name of the
${pvc.annotations['<ANNOTATION_KEY>']}
(e.g.${pvc.annotations['example.com/key']}
)- Replaced with the value of the specified annotation from the
PersistentVolumeClaim
object that triggered provisioning
- Replaced with the value of the specified annotation from the
csi.storage.k8s.io/controller-expand-secret-namespace
${pv.name}
- Replaced with name of the
PersistentVolume
object being provisioned.
- Replaced with name of the
${pvc.namespace}
- Replaced with namespace of the
PersistentVolumeClaim
object that triggered provisioning.
- Replaced with namespace of the
VolumeSnapshotClass Secrets
The CSI external-snapshotter sidecar container facilitates the handling of secrets for the following operations:
CreateSnapshotRequest
DeleteSnapshotRequest
CSI external-snapshotter
v1.0.1+ supports the following keys in VolumeSnapshotClass.parameters
:
csi.storage.k8s.io/snapshotter-secret-name
csi.storage.k8s.io/snapshotter-secret-namespace
Cluster admins can populate the secret fields for the operations listed above with data from Kubernetes Secret
objects by specifying these keys in the VolumeSnapshotClass
object.
Operations
Details for each secret supported by the external-snapshotter can be found below.
Create/Delete VolumeSnapshot Secret
CSI external-snapshotter
v1.0.1+ looks for the following keys in VolumeSnapshotClass.parameters
:
csi.storage.k8s.io/snapshotter-secret-name
csi.storage.k8s.io/snapshotter-secret-namespace
The values of both of these parameters, together, refer to the name and namespace of a Secret
object in the Kubernetes API.
If specified, the CSI external-snapshotter
will attempt to fetch the secret before creation and deletion.
If the secret is retrieved successfully, the snapshotter passes it to the CSI driver in the CreateSnapshotRequest.secrets
or DeleteSnapshotRequest.secrets
field.
If no such secret exists in the Kubernetes API, or the snapshotter is unable to fetch it, the create operation will fail.
Note, however, that the delete operation will continue even if the secret is not found (because, for example, the entire namespace containing the secret was deleted). In this case, if the driver requires a secret for deletion, then the volume and PV may need to be manually cleaned up.
The values of these parameters may be "templates". The external-snapshotter
will automatically resolve templates at snapshot create time, as detailed below:
csi.storage.k8s.io/snapshotter-secret-name
${volumesnapshotcontent.name}
- Replaced with name of the
VolumeSnapshotContent
object being created.
- Replaced with name of the
${volumesnapshot.namespace}
- Replaced with namespace of the
VolumeSnapshot
object that triggered creation.
- Replaced with namespace of the
${volumesnapshot.name}
- Replaced with the name of the
VolumeSnapshot
object that triggered creation.
- Replaced with the name of the
csi.storage.k8s.io/snapshotter-secret-namespace
${volumesnapshotcontent.name}
- Replaced with name of the
VolumeSnapshotContent
object being created.
- Replaced with name of the
${volumesnapshot.namespace}
- Replaced with namespace of the
VolumeSnapshot
object that triggered creation.
- Replaced with namespace of the
CSI Topology Feature
Status
Status | Min K8s Version | Max K8s Version | external-provisioner Version |
---|---|---|---|
Alpha | 1.12 | 1.12 | 0.4 |
Alpha | 1.13 | 1.13 | 1.0 |
Beta | 1.14 | - | 1.1+ |
Overview
Some storage systems expose volumes that are not equally accessible by all nodes in a Kubernetes cluster. Instead volumes may be constrained to some subset of node(s) in the cluster. The cluster may be segmented into, for example, “racks” or “regions” and “zones” or some other grouping, and a given volume may be accessible only from one of those groups.
To enable orchestration systems, like Kubernetes, to work well with storage systems which expose volumes that are not equally accessible by all nodes, the CSI spec enables:
- Ability for a CSI Driver to opaquely specify where a particular node exists (e.g. "node A" is in "zone 1").
- Ability for Kubernetes (users or components) to influence where a volume is provisioned (e.g. provision new volume in either "zone 1" or "zone 2").
- Ability for a CSI Driver to opaquely specify where a particular volume exists (e.g. "volume X" is accessible by all nodes in "zone 1" and "zone 2").
Kubernetes and the external-provisioner use these abilities to make intelligent scheduling and provisioning decisions (that Kubernetes can both influence and act on topology information for each volume),
Implementing Topology in your CSI Driver
To support topology in a CSI driver, the following must be implemented:
- The
PluginCapability
must supportVOLUME_ACCESSIBILITY_CONTRAINTS
. - The plugin must fill in
accessible_topology
inNodeGetInfoResponse
. This information will be used to populate the Kubernetes CSINode object and add the topology labels to the Node object. - During
CreateVolume
, the topology information will get passed in throughCreateVolumeRequest.accessibility_requirements
.
In the StorageClass object, both volumeBindingMode
values of Immediate
and
WaitForFirstConsumer
are supported.
- If
Immediate
is set, then the external-provisioner will pass in all available topologies in the cluster for the driver. - If
WaitForFirstConsumer
is set, then the external-provisioner will wait for the scheduler to pick a node. The topology of that selected node will then be set as the first entry inCreateVolumeRequest.accessibility_requirements.preferred
. All remaining topologies are still included in therequisite
andpreferred
fields to support storage systems that span across multiple topologies.
Sidecar Deployment
The topology feature requires the external-provisioner sidecar with the Topology feature gate enabled:
--feature-gates=Topology=true
Kubernetes Cluster Setup
Beta
In the Kubernetes cluster the CSINodeInfo
feature must be enabled on both Kubernetes master and nodes (refer to the CSINode Object section for more info):
--feature-gates=CSINodeInfo=true
In order to fully function properly, all Kubernetes master and nodes must be on at least
Kubernetes 1.14. If a selected node is on a lower version, topology is ignored and not
passed to the driver during CreateVolume
.
Alpha
The alpha feature in the external-provisioner is not compatible across Kubernetes versions. In addition, Kubernetes master and node version skew and upgrades are not supported.
The CSINodeInfo
, VolumeScheduling
, and KubeletPluginsWatcher
feature gates
must be enabled on both Kubernetes master and nodes.
The CSINodeInfo CRDs also have to be manually installed in the cluster.
Storage Internal Topology
Note that a storage system may also have an "internal topology" different from (independent of) the topology of the cluster where workloads are scheduled. Meaning volumes exposed by the storage system are equally accessible by all nodes in the Kubernetes cluster, but the storage system has some internal topology that may influence, for example, the performance of a volume from a given node.
CSI does not currently expose a first class mechanism to influence such storage system internal topology on provisioning. Therefore, Kubernetes can not programmatically influence such topology. However, a CSI Driver may expose the ability to specify internal storage topology during volume provisioning using an opaque parameter in the CreateVolume
CSI call (CSI enables CSI Drivers to expose an arbitrary set of configuration options during dynamic provisioning by allowing opaque parameters to be passed from cluster admins to the storage plugins) -- this would enable cluster admins to be able to control the storage system internal topology during provisioning.
# Raw Block Volume Feature
Status
Status | Min K8s Version | Max K8s Version | external-provisioner Version | external-attacher Version |
---|---|---|---|---|
Alpha | 1.11 | 1.13 | 0.4 | 0.4 |
Alpha | 1.13 | 1.13 | 1.0 | 1.0 |
Beta | 1.14 | - | 1.1+ | 1.1+ |
Overview
This page documents how to implement raw block volume support to a CSI Driver.
A block volume is a volume that will appear as a block device inside the container. A mounted (file) volume is volume that will be mounted using a specified file system and appear as a directory inside the container.
The CSI spec supports both block and mounted (file) volumes.
Implementing Raw Block Volume Support in Your CSI Driver
CSI doesn't provide a capability query for block volumes, so COs will simply pass through requests for
block volume creation to CSI plugins, and plugins are allowed to fail with the InvalidArgument
GRPC
error code if they don't support block volumes. Kubernetes doesn't make any assumptions about which CSI
plugins support blocks and which don't, so users have to know if any given storage class is capable of
creating block volumes.
The difference between a request for a mounted (file) volume and a block volume is the VolumeCapabilities
field of the request. Note that this field is an array and the created volume must support ALL of the
capabilities requested, or else return an error. If the AccessType
method of a VolumeCapability
VolumeCapability_Block
, then the capability is requesting a raw block volume. Unlike mount volumes, block
volumes don't have any specific capabilities that need to be validated, although access modes still
apply.
Block volumes are much more likely to support multi-node flavors of VolumeCapability_AccessMode_Mode
than mount volumes, because there's no file system state stored on the node side that creates any technical
impediments to multi-attaching block volumes. While there may still be good reasons to prevent
multi-attaching block volumes, and there may be implementations that are not capable of supporting
multi-attach, you should think carefully about what makes sense for your driver.
CSI plugins that support both mount and block volumes must be sure to check the capabilities of all CSI RPC requests and ensure that the capability of the request matches the capability of the volume, to avoid trying to do file-system-related things to block volumes and block-related things to file system volumes. The following RPCs specify capabilities that must be validated:
CreateVolume()
(multiple capabilities)ControllerPublishVolume()
ValidateVolumeCapabilities()
(multiple capabilities)GetCapacity()
(see below)NodeStageVolume()
NodePublishVolume()
Also, CSI plugins that implement the optional GetCapacity()
RPC should note that that RPC includes
capabilities too, and if the capacity for mount volumes is not the same as the capacity for block
volumes, that needs to be handled in the implementation of that RPC.
Q: Can CSI plugins support only block volumes and not mount volumes?
A: Yes! This is just the reverse case of supporting mount volumes only. Plugins may return InvalidArgument
for any creation request with an AccessType
of VolumeCapability_Mount
.
Differences Between Block and Mount Volumes
The main difference between block volumes and mount volumes is the expected result of the NodePublish()
.
For mount volumes, the CO expects the result to be a mounted directory, at TargetPath
. For block volumes,
the CO expects there to be a device file at TargetPath
. The device file can by a bind-mounted device from
the hosts /dev
file system, or it can be a device node created at that location using mknod()
.
It's desirable but not required to expose an unfiltered device node. For example, CSI plugins based on technologies that implement SCSI protocols should expect that pods consuming the block volumes they create may want to send SCSI commands to the device. This is something that should "just work" by default (subject to container capabilities) so CSI plugins should avoid anything that would break this kind of use case. The only hard requirement is that the device implements block reading/writing however.
For plugins with the RPC_STAGE_UNSTAGE_VOLUME
capability, the CO doesn't care exactly what is placed at
the StagingTargetPath
, but it's worth noting that some CSI RPCs are allowed to pass the plugin either
a staging path or a publish path, so it's important to think carefully about how NodeStageVolume()
is
implemented, knowing that either path could get used by the CO to refer to the volume later on. This is
made more challenging because the CSI spec says that StagingTargetPath
is always a directory even for
block volumes.
Sidecar Deployment
The raw block feature requires the external-provisioner and external-attacher sidecars to be deployed.
Kubernetes Cluster Setup
The BlockVolume
and CSIBlockVolume
feature gates need to be enabled on
all Kubernetes masters and nodes.
--feature-gates=BlockVolume=true,CSIBlockVolume=true...
- TODO: detail how Kubernetes API raw block fields get mapped to CSI methods/fields.
Skip Kubernetes Attach and Detach
Status
Status | Min K8s Version | Max K8s Version | cluster-driver-registrar Version |
---|---|---|---|
Alpha | 1.12 | 1.12 | 0.4 |
Alpha | 1.13 | 1.13 | 1.0 |
Beta | 1.14 | - | n/a |
Overview
Volume drivers, like NFS, for example, have no concept of an attach (ControllerPublishVolume
). However, Kubernetes always executes Attach
and Detach
operations even if the CSI driver does not implement an attach operation (i.e. even if the CSI Driver does not implement a ControllerPublishVolume
call).
This was problematic because it meant all CSI drivers had to handle Kubernetes attachment. CSI Drivers that did not implement the PUBLISH_UNPUBLISH_VOLUME
controller capability could work around this by deploying an external-attacher and the external-attacher
would responds to Kubernetes attach operations and simply do a noop (because the CSI driver did not advertise the PUBLISH_UNPUBLISH_VOLUME
controller capability).
Although the workaround works, it adds an unnecessary operation (round-trip) in the preparation of a volume for a container, and requires CSI Drivers to deploy an unnecessary sidecar container (external-attacher
).
Skip Attach with CSI Driver Object
The CSIDriver Object enables CSI Drivers to specify how Kubernetes should interact with it.
Specifically the attachRequired
field instructs Kubernetes to skip any attach operation altogether.
For example, the existence of the following object would cause Kubernetes to skip attach operations for all CSI Driver testcsidriver.example.com
volumes.
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: testcsidriver.example.com
spec:
attachRequired: false
CSIDriver object should be manually included in the driver deployment manifests.
Previously, the cluster-driver-registrar sidecar container could be deployed to automatically create the object. Once the flags to this container are configured correctly, it will automatically create a CSIDriver Object when it starts with the correct fields set.
Alpha Functionality
In alpha, this feature was enabled via the CSIDriver Object CRD.
apiVersion: csi.storage.k8s.io/v1alpha1
kind: CSIDriver
metadata:
....
Pod Info on Mount
Status
Status | Min K8s Version | Max K8s Version | cluster-driver-registrar Version |
---|---|---|---|
Alpha | 1.12 | 1.12 | 0.4 |
Alpha | 1.13 | 1.13 | 1.0 |
Beta | 1.14 | - | n/a |
Overview
CSI avoids encoding Kubernetes specific information in to the specification, since it aims to support multiple orchestration systems (beyond just Kubernetes).
This can be problematic because some CSI drivers require information about the workload (e.g. which pod is referencing this volume), and CSI does not provide this information natively to drivers.
Pod Info on Mount with CSI Driver Object
The CSIDriver Object enables CSI Drivers to specify how Kubernetes should interact with it.
Specifically the podInfoOnMount
field instructs Kubernetes that the CSI driver requires additional pod information (like podName, podUID, etc.) during mount operations.
For example, the existence of the following object would cause Kubernetes to add pod information at mount time to the NodePublishVolumeRequest.publish_context
map.
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: testcsidriver.example.com
spec:
podInfoOnMount: true
If the podInfoOnMount
field is set to true
, during mount, Kubelet will add the following key/values to the publish_context
field in the CSI NodePublishVolumeRequest
:
csi.storage.k8s.io/pod.name: {pod.Name}
csi.storage.k8s.io/pod.namespace: {pod.Namespace}
csi.storage.k8s.io/pod.uid: {pod.UID}
The CSIDriver object should be manually included in the driver manifests.
Previously, the cluster-driver-registrar sidecar container could be used to create the object. Once the flags to this container are configured correctly, it will automatically create a CSIDriver Object when it starts with the correct fields set.
Alpha Functionality
In alpha, this feature was enabled by setting the podInfoOnMountVersion
field in the CSIDriver
Object CRD to to v1
.
apiVersion: csi.storage.k8s.io/v1alpha1
kind: CSIDriver
metadata:
name: testcsidriver.example.com
spec:
podInfoOnMountVersion: v1
Volume Expansion
Status
Status | Min K8s Version | Max K8s Version | external-resizer Version |
---|---|---|---|
Alpha | 1.14 | 1.15 | 0.2 |
Beta | 1.16 | - | 0.3 |
Overview
A storage provider that allows volume expansion after creation, may choose to implement volume expansion either via a control-plane CSI RPC call or via node CSI RPC call or both as a two step process.
Implementing Volume expansion functionality
To implement volume expansion the CSI driver MUST:
- Implement
VolumeExpansion
plugin capability. - Implement
EXPAND_VOLUME
controller capability or implementEXPAND_VOLUME
node capability or both.
ControllerExpandVolume
RPC call can be made when volume is ONLINE
or OFFLINE
depending on VolumeExpansion
plugin
capability. Where ONLINE
and OFFLINE
means:
- ONLINE : Volume is currently published or available on a node.
- OFFLINE : Volume is currently not published or available on a node.
NodeExpandVolume
RPC call on the other hand - always requires volume to be published or staged on a node (and hence ONLINE
).
For block storage file systems - NodeExpandVolume
is typically used for expanding the file system on the node, but it can be also
used to perform other volume expansion related housekeeping operations on the node.
For details, see the CSI spec.
Deploying volume expansion functionality
The Kubernetes CSI development team maintains external-resizer Kubernetes CSI Sidecar Containers.
This sidecar container implements the logic for watching the Kubernetes API for Persistent Volume claim edits and issuing ControllerExpandVolume
RPC call against a CSI endpoint and updating PersistentVolume
object to reflect new size.
This sidecar is needed even if CSI driver does not have EXPAND_VOLUME
controller capability, in this case it performs a NO-OP expansion and updates PersistentVolume
object. NodeExpandVolume
is always called by Kubelet on the node.
For more details, see external-resizer.
Enabling Volume expansion for CSI volumes in Kubernetes
To expand a volume if permitted by the storage class, users just need to edit the persistent volume claim object and request more storage.
In Kubernetes 1.14 and 1.15, this feature was in alpha status and required enabling the following feature gate:
--feature-gates=ExpandCSIVolumes=true
Also in Kubernetes 1.14 and 1.15, online expansion had to be enabled explicitly:
--feature-gates=ExpandInUsePersistentVolumes=true
external-resizer and kubelet add appropriate events and conditions to persistent volume claim objects indicating progress of volume expansion operations.
Kubernetes PVC DataSource (CSI VolumeContentSource)
When creating a new PersistentVolumeClaim, the Kubernetes API provides a PersistentVolumeClaim.DataSource
parameter. This parameter is used to specify the CSI CreateVolumeRequest.VolumeContentSource
option for CSI Provisioners. The VolumeContentSource
parameter instructs the CSI plugin to pre-populate the volume being provisioned with data from the specified source.
External Provisioner Responsibilities
If a DataSource
is specified in the CreateVolume
call to the CSI external provisioner, the external provisioner will fetch the specified resource and pass the appropriate object id to the plugin.
Supported DataSources
Currently there are two types of PersistentVolumeClaim.DataSource
objects that are supported:
Volume Cloning
Status and Releases
Status | Min k8s Version | Max k8s version | external-provisioner Version |
---|---|---|---|
Alpha | 1.15 | 1.15 | 1.3 |
Beta | 1.16 | - | 1.4 |
Overview
A Clone is defined as a duplicate of an existing Kubernetes Volume. For more information on cloning in Kubernetes see the concepts doc for Volume Cloning. A storage provider that allows volume cloning as a create feature, may choose to implement volume cloning via a control-plan CSI RPC call.
For details regarding the kuberentes API for volume cloning, please see kubernetes concepts.
Implementing Volume cloning functionality
To implement volume cloning the CSI driver MUST:
- Implement checks for
csi.CreateVolumeRequest.VolumeContentSource
in the plugin'sCreateVolume
function implementation. - Implement
CLONE_VOLUME
controller capability.
It is the responsibility of the storage plugin to either implement an expansion after clone if a provision request size is greater than the source, or allow the external-resizer to handle it. In the case that the plugin does not support resize capability and it does not have the capability to create a clone that is greater in size than the specified source volume, then the provision request should result in a failure.
Deploying volume clone functionality
The Kubernetes CSI development team maintains the external-provisioner which is responsible for detecting requests for a PVC DataSource and providing that information to the plugin via the csi.CreateVolumeRequest
. It's up to the plugin to check the csi.CreateVolumeRequest
for a VolumeContentSource
entry in the CreateVolumeRequest object.
There are no additional side-cars or add on components required.
Enabling Cloning for CSI volumes in Kubernetes
In Kubernetes 1.15 this feature was alpha status and required enabling the appropriate feature gate:
--feature-gates=VolumePVCDataSource=true
Example implementation
A trivial example implementation can be found in the csi-hostpath plugin in its implementation of CreateVolume
.
Snapshot & Restore Feature
Status
Status | Min K8s Version | Max K8s Version | external-snapshotter Version | external-provisioner Version |
---|---|---|---|---|
Alpha | 1.12 | 1.12 | 0.4 | 0.4 |
Alpha | 1.13 | - | 1.0+ | 1.0+ |
Overview
Many storage systems provide the ability to create a "snapshot" of a persistent volume. A snapshot represents a point-in-time copy of a volume. A snapshot can be used either to provision a new volume (pre-populated with the snapshot data) or to restore the existing volume to a previous state (represented by the snapshot).
Kubernetes CSI currently enables CSI Drivers to expose the following functionality via the Kubernetes API:
- Creation and deletion of volume snapshots via Kubernetes native API.
- Creation of new volumes pre-populated with the data from a snapshot via Kubernetes dynamic volume provisioning.
Implementing Snapshot & Restore Functionality in Your CSI Driver
To implement the snapshot feature, a CSI driver MUST:
- Implement the
CREATE_DELETE_SNAPSHOT
and, optionally, theLIST_SNAPSHOTS
controller capabilities - Implement
CreateSnapshot
,DeleteSnapshot
, and, optionally, theListSnapshots
, controller RPCs.
For details, see the CSI spec.
Sidecar Deployment
The Kubernetes CSI development team maintains the external-snapshotter Kubernetes CSI Sidecar Containers. This sidecar container implements the logic for watching the Kubernetes API for snapshot objects and issuing the appropriate CSI snapshot calls against a CSI endpoint. For more details, see external-snapshotter documentation.
Snapshot APIs
Similar to the API for managing Kubernetes Persistent Volumes, the Kubernetes Volume Snapshots introduce three new API objects for managing snapshots: VolumeSnapshot
, VolumeSnapshotContent
, and VolumeSnapshotClass
. See Kubernetes Snapshot documentation for more details.
Unlike the core Kubernetes Persistent Volume objects, these Snapshot objects are defined as Custom Resource Definitions (CRDs). This is because the Kubernetes project is moving away from having resource types pre-defined in the API server. This allows the API server to be reused for projects other than Kubernetes, and consumers (like Kubernetes) simply install the resource types they require as CRDs. Because the Snapshot API types are not built in to Kubernetes, they must be installed prior to use.
The CRDs are automatically deployed by the CSI external-snapshotter sidecar.
The schema definition for the custom resources (CRs) can be found here: https://github.com/kubernetes-csi/external-snapshotter/blob/master/pkg/apis/volumesnapshot/v1alpha1/types.go
In addition to these new CRD objects, a new, alpha DataSource
field has been added to the PersistentVolumeClaim
object. This new field enables dynamic provisioning of new volumes that are automatically pre-populated with data from an existing snapshot.
Kubernetes Cluster Setup
Since volume snapshot is an alpha feature in Kubernetes v1.12, you need to enable a new alpha feature gate called VolumeSnapshotDataSource
in the Kubernetes master.
--feature-gates=VolumeSnapshotDataSource=true
Test Snapshot Feature
Use the following example yaml files to test the snapshot feature.
Create a StorageClass:
kubectl create -f storageclass.yaml
Create a PVC:
kubectl create -f pvc.yaml
Create a VolumeSnapshotClass:
kubectl create -f snapshotclass.yaml
Create a VolumeSnapshot:
kubectl create -f snapshot.yaml
Create a PVC from a VolumeSnapshot:
kuberctl create -f restore.yaml
PersistentVolumeClaim not Bound
If a PersistentVolumeClaim
is not bound, the attempt to create a volume snapshot from that PersistentVolumeClaim
will fail. No retries will be attempted. An event will be logged to indicate that the PersistentVolumeClaim
is not bound.
Note that this could happen if the PersistentVolumeClaim
spec and the VolumeSnapshot
spec are in the same YAML file. In this case, when the VolumeSnapshot
object is created, the PersistentVolumeClaim
object is created but volume creation is not complete and therefore the PersistentVolumeClaim
is not yet bound. You must wait until the PersistentVolumeClaim
is bound and then create the snapshot.
Examples
See the Drivers for a list of CSI drivers that implement the snapshot feature.
Pod Inline Volume Support
Status
Status | Min K8s Version | Max K8s Version |
---|---|---|
Alpha | 1.15 | 1.15 |
Beta | 1.16 | - |
Overview
Traditionally, volumes that are backed by CSI drivers can only be used with a PersistentVolume
and PersistentVolumeClaim
object combination. This feature supports ephemeral storage use cases and allows CSI volumes to be specified directly in the pod specification. At runtime, nested inline volumes follow the ephemeral lifecycle of their associated pods where the driver handles all phases of volume operations as pods are created and destroyed.
See the design document for futher information.
Example of inline CSI pod spec
A pod spec with an ephemeral inline CSI volume. Note that because the volume is expected to be ephemeral, the volumeHandle is not provided. Instead, an ID will be generated by kubelet as volume is mounted for pod use. The generated ID is passed to the CSI driver at NodePublish
.
apiVersion: v1
kind: Pod
metadata:
name: some-pod
spec:
containers:
...
volumes:
- name: vol
csi:
driver: inline.storage.kubernetes.io
volumeAttributes:
foo: bar
Implementing inline ephemeral support
Drivers must be modified (or implemented specifically) to support inline ephemeral workflows. When Kubernetes encounters an inline CSI volume embedded in a pod spec, it treats that volume differently. Mainly, the driver will only receive NodePublish
, during the volume's mount phase, and NodeUnpublish
when the pod is going away and the volume is unmounted. To support inline, a driver must implement the followings:
- Identity service
- Node service
Kubernetes 1.16 only allows using a CSI driver for an inline volume if
its CSIDriverInfo
object explicitly declares
that the driver supports that kind of usage in its
volumeLifecycleModes
field. This is a safeguard against accidentally
using a driver the wrong way.
Feature gates
To use inline volume, Kubernetes 1.15 binaries must start with the CSIInlineVolume
feature gate enabled:
--feature-gates=CSIInlineVolume=true
Kubernetes >= 1.16 no longer needs this as the feature is enabled by default.
Example implementation
- CSI Hostpath driver - an example driver that supports both modes and determines the mode on a case-by-case basis (for Kubernetes 1.16) or can be deployed with support for just one of the two modes (for Kubernetes 1.15).
- Image populator plugin - an example CSI driver plugin that uses a container image as a volume.
Volume Limits
Status
- Kubernetes 1.11 - 1.11: Alpha
- Kubernetes 1.12 - 1.16: Beta
- Kubernetes 1.17: GA
Overview
Some storage providers may have a restriction on the number of volumes that can be used in a Node. This is common in cloud providers, but other providers might impose restriction as well.
Kubernetes will respect this limit as long the CSI driver advertises it. To support volume limits in a CSI driver, the plugin must fill in max_volumes_per_node
in NodeGetInfoResponse
.
It is recommended that CSI drivers allow for customization of volume limits. That way cluster administrators can distribute the limits of the same storage backends (e.g. iSCSI) accross different drivers, according to their individual needs.
Deploying CSI Driver on Kubernetes
This page describes to CSI driver developers how to deploy their driver onto a Kubernetes cluster.
Overview
A CSI driver is typically deployed in Kubernetes as two components: a controller component and a per-node component.
Controller Plugin
The controller component can be deployed as a Deployment or StatefulSet on any node in the cluster. It consists of the CSI driver that implements the CSI Controller service and one or more sidecar containers. These controller sidecar containers typically interact with Kubernetes objects and make calls to the driver's CSI Controller service.
It generally does not need direct access to the host and can perform all its operations through the Kubernetes API and external control plane services. Multiple copies of the controller component can be deployed for HA, however it is recommended to use leader election to ensure there is only one active controller at a time.
Controller sidecars include the external-provisioner, external-attacher, external-snapshotter, and external-resizer. Including a sidecar in the deployment may be optional. See each sidecar's page for more details.
Communication with Sidecars
Sidecar containers manage Kubernetes events and make the appropriate calls to the CSI driver. The calls are made by sharing a UNIX domain socket through an emptyDir volume between the sidecars and CSI Driver.
RBAC Rules
Most controller sidecars interact with Kubernetes objects and therefore need to set RBAC policies. Each sidecar repository contains example RBAC configurations.
Node Plugin
The node component should be deployed on every node in the cluster through a DaemonSet. It consists of the CSI driver that implements the CSI Node service and the node-driver-registrar sidecar container.
Communication with Kubelet
The Kubernetes kubelet runs on every node and is responsible for making the CSI Node service calls. These calls mount and unmount the storage volume from the storage system, making it available to the Pod to consume. Kubelet makes calls to the CSI driver through a UNIX domain socket shared on the host via a HostPath volume. There is also a second UNIX domain socket that the node-driver-registrar uses to register the CSI driver to kubelet.
Driver Volume Mounts
The node plugin needs direct access to the host for making block devices and/or filesystem mounts available to the Kubernetes kubelet.
The mount point used by the CSI driver must be set to Bidirectional to allow Kubelet on the host to see mounts created by the CSI driver container. See the example below:
containers:
- name: my-csi-driver
...
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: node-driver-registrar
...
volumeMounts:
- name: registration-dir
mountPath: /registration
volumes:
# This volume is where the socket for kubelet->driver communication is done
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/<driver-name>
type: DirectoryOrCreate
# This volume is where the driver mounts volumes
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
# This volume is where the node-driver-registrar registers the plugin
# with kubelet
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
Deploying
Deploying a CSI driver onto Kubernetes is highlighted in detail in Recommended Mechanism for Deploying CSI Drivers on Kubernetes.
Enable privileged Pods
To use CSI drivers, your Kubernetes cluster must allow privileged pods (i.e. --allow-privileged
flag must be set to true
for both the API server and the kubelet). This is the default in some environments (e.g. GCE, GKE, kubeadm
).
Ensure your API server are started with the privileged flag:
$ ./kube-apiserver ... --allow-privileged=true ...
$ ./kubelet ... --allow-privileged=true ...
Note: Starting from Kubernetes 1.13.0, --allow-privileged is true for kubelet. It'll be deprecated in future kubernetes releases.
Enabling mount propagation
Another feature that CSI depends on is mount propagation. It allows the sharing of volumes mounted by one container with other containers in the same pod, or even to other pods on the same node. For mount propagation to work, the Docker daemon for the cluster must allow shared mounts. See the mount propagation docs to find out how to enable this feature for your cluster. This page explains how to check if shared mounts are enabled and how to configure Docker for shared mounts.
Examples
- Simple deployment example using a single pod for all components: see the hostpath example.
- Full deployment example using a DaemonSet for the node plugin and StatefulSet for the controller plugin: TODO
More information
For more information, please read CSI Volume Plugins in Kubernetes Design Doc.
Example
The Hostpath CSI driver is a simple sample driver that provisions a directory on the host. It can be used as an example to get started writing a driver, however it is not meant for production use. The deployment example shows how to deploy and use that driver in Kubernetes.
The example deployment uses the original RBAC rule files that are maintained together with sidecar apps and deploys into the default namespace. A real production should copy the RBAC files and customize them as explained in the comments of those files.
If you encounter any problems, please check the Troubleshooting page.
Testing
This section describes how CSI developers can test their CSI drivers.
Unit Testing
The CSI sanity
package from csi-test can be used for unit testing your CSI driver.
It contains a set of basic tests that all CSI drivers should pass (for example, NodePublishVolume should fail when no volume id is provided
, etc.).
This package can be used in two modes:
- Via a Golang test framework (
sanity
package is imported as a dependency) - Via a command line against your driver binary.
Read the documentation of the sanity
package for more details.
Functional Testing
Drivers should be functionally "end-to-end" tested while deployed in a Kubernetes cluster. Previously, how to do this and what tests to run was left up to driver authors. Now, a standard set of Kubernetes CSI end-to-end tests can be imported and run by third party CSI drivers. This documentation specifies how to do so.
The CSI community is also looking in to establishing an official "CSI Conformance Suite" to recognize "officially certified CSI drivers". This documentation will be updated with more information once that process has been defined.
Kubernetes End to End Testing for CSI Storage Plugins
Currently, csi-sanity exists to help test compliance with the CSI spec, but e2e testing of plugins is needed as well to provide plugin authors and users validation that their plugin is integrated well with specific versions of Kubernetes.
Setting up End to End tests for your CSI Plugin
Prerequisites:
- A Kubernetes v1.13+ Cluster
- Kubectl
There are two ways to run end-to-end tests for your CSI Plugin
- use Kubernetes E2E Tests, by providing a DriverDefinition YAML file via a parameter.
- Note: In some cases you would not be able to use this method, in running e2e tests by just providing a YAML file defining your CSI plugin. For example the NFS CSI plugin currently does not support dynamic provisoning, so we would want to skip those and run only pre-provisioned tests. For such cases, you would need to write your own testdriver, which is discussed below.
- import the in-tree storage tests and run them using
go test
.
This doc will cover how to run the E2E tests using the second method.
Importing the E2E test suite as a library
In-tree storage e2e tests could be used to test CSI storage plugins. Your repo should be setup similar to how the NFS CSI plugin is setup, where the testfiles are in a test
directory and the main test file is in the cmd
directory.
To be able to import Kubernetes in-tree storage tests, the CSI plugin would need to use Kubernetes v1.14+ (add to plugin's GoPkg.toml, since pluggable E2E tests become available in v1.14). CSI plugin authors would also be required to implement a testdriver for their CSI plugin. The testdriver provides required functionality that would help setup testcases for a particular plugin.
For any testdriver these functions would be required (Since it implements the TestDriver Interface):
GetDriverInfo() *testsuites.DriverInfo
SkipUnsupportedTest(pattern testpatterns.TestPattern)
PrepareTest(f *framework.Framework) (*testsuites.PerTestConfig, func())
The PrepareTest
method is where you would write code to setup your CSI plugin, and it would be called before each test case. It is recommended that you don't deploy your plugin in this method, and rather deploy it manually before running your tests.
GetDriverInfo
will return a DriverInfo
object that has all of the plugin's capabilities and required information. This object helps tests find the deployed plugin, and also decides which tests should run (depending on the plugin's capabilities).
Here are examples of the NFS and Hostpath DriverInfo objects:
testsuites.DriverInfo{
Name: "csi-nfsplugin",
MaxFileSize: testpatterns.FileSizeLarge,
SupportedFsType: sets.NewString(
"", // Default fsType
),
Capabilities: map[testsuites.Capability]bool{
testsuites.CapPersistence: true,
testsuites.CapExec: true,
},
}
testsuites.DriverInfo{
Name: "csi-hostpath",
FeatureTag: "",
MaxFileSize: testpatterns.FileSizeMedium,
SupportedFsType: sets.NewString(
"", // Default fsType
),
Capabilities: map[testsuites.Capability]bool{
testsuites.CapPersistence: true,
},
}
You would define something similar for your CSI plugin.
SkipUnsupportedTest
simply skips any tests that you define there.
Depending on your plugin's specs, you would implement other interaces defined here. For example the NFS testdriver also implements PreprovisionedVolumeTestDriver and PreprovisionedPVTestDriver interfaces, to enable pre-provisoned tests.
After implementing the testdriver for your CSI plugin, you would create a csi-volumes.go
file, where the implemented testdriver is used to run in-tree storage testsuites, similar to how the NFS CSI plugin does so. This is where you would define which testsuites you would want to run for your plugin. All available in-tree testsuites can be found here.
Finally, importing the test
package into your main test file will initialize the testsuites to run the E2E tests.
The NFS plugin creates a binary to run E2E tests, but you could use go test
instead to run E2E tests using a command like this:
go test -v <main test file> -ginkgo.v -ginkgo.progress --kubeconfig=<kubeconfig file> -timeout=0
Drivers
The following are a set of CSI driver which can be used with Kubernetes:
NOTE: If you would like your driver to be added to this table, please open a pull request in this repo updating this file. Other Features is allowed to be filled in Raw Block, Snapshot, Expansion and Cloning. If driver did not implement any Other Features, please leave it blank.
Production Drivers
Name | CSI Driver Name | Compatible with CSI Version(s) | Description | Persistence (Beyond Pod Lifetime) | Supported Access Modes | Dynamic Provisioning | Other Features |
---|---|---|---|---|---|---|---|
Alicloud Disk | diskplugin.csi.alibabacloud.com | v1.0 | A Container Storage Interface (CSI) Driver for Alicloud Disk | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot |
Alicloud NAS | nasplugin.csi.alibabacloud.com | v1.0 | A Container Storage Interface (CSI) Driver for Alicloud Network Attached Storage (NAS) | Persistent | Read/Write Multiple Pods | No | |
Alicloud OSS | ossplugin.csi.alibabacloud.com | v1.0 | A Container Storage Interface (CSI) Driver for Alicloud Object Storage Service (OSS) | Persistent | Read/Write Multiple Pods | No | |
AWS Elastic Block Storage | ebs.csi.aws.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for AWS Elastic Block Storage (EBS) | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot, Expansion |
AWS Elastic File System | efs.csi.aws.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for AWS Elastic File System (EFS) | Persistent | Read/Write Multiple Pods | No | |
AWS FSx for Lustre | fsx.csi.aws.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for AWS FSx for Lustre (EBS) | Persistent | Read/Write Multiple Pods | Yes | |
Azure disk | disk.csi.azure.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Azure disk | Persistent | Read/Write Single Pod | Yes | |
Azure file | file.csi.azure.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Azure file | Persistent | Read/Write Multiple Pods | Yes | |
CephFS | cephfs.csi.ceph.com | v0.3, v1.0.0, v1.1.0 | A Container Storage Interface (CSI) Driver for CephFS | Persistent | Read/Write Multiple Pods | Yes | |
Ceph RBD | rbd.csi.ceph.com | v0.3, v1.0.0, v1.1.0 | A Container Storage Interface (CSI) Driver for Ceph RBD | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot |
ChubaoFS | csi.chubaofs.com | v1.0.0 | A Container Storage Interface (CSI) Driver for ChubaoFS Storage | Persistent | Read/Write Multiple Pods | Yes | Raw Block |
Cinder | cinder.csi.openstack.org | v0.3, v1.0, v1.1 | A Container Storage Interface (CSI) Driver for OpenStack Cinder | Persistent and Ephemeral | Depends on the storage backend used | Yes, if storage backend supports it | Raw Block, Snapshot, Expansion |
cloudscale.ch | csi.cloudscale.ch | v1.0 | A Container Storage Interface (CSI) Driver for the cloudscale.ch IaaS platform | Persistent | Read/Write Single Pod | Yes | Snapshot |
Datera | dsp.csi.daterainc.io | v1.0 | A Container Storage Interface (CSI) Driver for Datera Data Services Platform (DSP) | Persistent | Read/Write Single Pod | Yes | Snapshot |
Dell EMC Isilon | csi-isilon.dellemc.com | v1.0 | A Container Storage Interface (CSI) Driver for Dell EMC Isilon | Persistent | Read/Write Multiple Pods | Yes | Snapshot |
Dell EMC PowerMax | csi-powermax.dellemc.com | v1.0 | A Container Storage Interface (CSI) Driver for Dell EMC PowerMax | Persistent | Read/Write Single Pod | Yes | |
Dell EMC Unity | csi-unity.dellemc.com | v1.0 | A Container Storage Interface (CSI) Driver for Dell EMC Unity | Persistent | Read/Write Single Pod | Yes | Snapshot |
Dell EMC VxFlexOS | csi-vxflexos.dellemc.com | v1.0 | A Container Storage Interface (CSI) Driver for Dell EMC VxFlexOS | Persistent | Read/Write Single Pod | Yes | Snapshot |
Dell EMC XtremIO | csi-xtremio.dellemc.com | v1.0 | A Container Storage Interface (CSI) Driver for Dell EMC XtremIO | Persistent | Read/Write Single Pod | Yes | Snapshot |
DigitalOcean Block Storage | dobs.csi.digitalocean.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage | Persistent | Read/Write Single Pod | Yes | Snapshot |
DriveScale | csi.drivescale.com | v1.0 | A Container Storage Interface (CSI) Driver for DriveScale software composable infrastructure solution | Persistent | Read/Write Single Pod | Yes | |
Ember CSI | [x].ember-csi.io | v0.2, v0.3, v1.0 | Multi-vendor CSI plugin supporting over 80 Drivers to provide block and mount storage to Container Orchestration systems. | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot |
GCE Persistent Disk | pd.csi.storage.gke.io | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Google Compute Engine Persistent Disk (GCE PD) | Persistent | Read/Write Single Pod | Yes | Snapshot |
Google Cloud Filestore | com.google.csi.filestore | v0.3 | A Container Storage Interface (CSI) Driver for Google Cloud Filestore | Persistent | Read/Write Multiple Pods | Yes | |
GlusterFS | org.gluster.glusterfs | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for GlusterFS | Persistent | Read/Write Multiple Pods | Yes | Snapshot |
Gluster VirtBlock | org.gluster.glustervirtblock | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Gluster Virtual Block volumes | Persistent | Read/Write Single Pod | Yes | |
Hammerspace CSI | com.hammerspace.csi | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Hammerspace Storage | Persistent | Read/Write Multiple Pods | Yes | Raw Block, Snapshot |
Hitachi Vantara | com.hitachi.hspc.csi | v1.0 | A Container Storage Interface (CSI) Driver for VSP series Storage | Persistent | ? | ? | |
HPE | csi.hpe.com | v0.3, v1.0, v1.1 | A Container Storage Interface (CSI) driver from HPE | Persistent or Ephemeral | Read/Write Single Pod | Yes | Raw Block, Snapshot, Expansion, Cloning |
IBM Block Storage | block.csi.ibm.com | v1.0, v1.1 | A Container Storage Interface (CSI) Driver for IBM FlashSystem 9100, IBM Spectrum Virtualize, IBM Storwize, IBM FlashSystem A9000 and A9000R. | Persistent | Read/Write Single Pod | Yes | |
Intel PMEM-CSI | pmem-csi.intel.com | v1.0 | A Container Storage Interface (CSI) driver for PMEM from Intel | Persistent | Read/Write Single Pod | Yes | Raw Block |
JuiceFS | csi.juicefs.com | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for JuiceFS File System | Persistent | Read/Write Multiple Pod | Yes | |
kaDalu | org.kadalu.gluster | v0.3 | A CSI Driver (and operator) for GlusterFS | Persistent | Read/Write Multiple Pods | Yes | |
Linode Block Storage | linodebs.csi.linode.com | v1.0 | A Container Storage Interface (CSI) Driver for Linode Block Storage | Persistent | Read/Write Single Pod | Yes | |
LINSTOR | io.drbd.linstor-csi | v1.1 | A Container Storage Interface (CSI) Driver for LINSTOR volumes | Persistent | Read/Write Single Pod | Yes | Snapshot |
Longhorn | driver.longhorn.io | v1.1 | A Container Storage Interface (CSI) Driver for Longhorn volumes | Persistent | Read/Write Single Node | Yes | Raw Block |
MacroSAN | csi-macrosan | v1.0 | A Container Storage Interface (CSI) Driver for MacroSAN Block Storage | Persistent | Read/Write Single Pod | Yes | |
MapR | com.mapr.csi-kdf | v1.0 | A Container Storage Interface (CSI) Driver for MapR Data Platform | Persistent | Read/Write Multiple Pods | Yes | Snapshot |
MooseFS | com.tuxera.csi.moosefs | v1.0 | A Container Storage Interface (CSI) Driver for MooseFS clusters. | Persistent | Read/Write Multiple Pods | Yes | |
NetApp | csi.trident.netapp.io | v1.0, v1.1 | A Container Storage Interface (CSI) Driver for NetApp's Trident container storage orchestrator | Persistent | Read/Write Multiple Pods | Yes | Snapshot, Expansion, Cloning |
NexentaStor | nexentastor-csi-driver.nexenta.com | v1.0 | A Container Storage Interface (CSI) Driver for NexentaStor | Persistent | Read/Write Multiple Pods | Yes | Snapshot |
Nutanix | "com.nutanix.csi" | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for Nutanix | Persistent | "Read/Write Single Pod" with Nutanix Volumes and "Read/Write Multiple Pods" with Nutanix Files | Yes | |
OpenEBS | cstor.csi.openebs.io | v1.0 | A Container Storage Interface (CSI) Driver for OpenEBS | Persistent | Read/Write Single Pod | Yes | Expansion, Snapshot, Cloning |
OpenSDS | csi-opensdsplugin | v1.0 | A Container Storage Interface (CSI) Driver for OpenSDS | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot |
Open-E | com.open-e.joviandss.csi | v1.0 | A Container Storage Interface (CSI) Driver for Open-E JovianDSS Storage | Persistent | Read/Write Single Pod | Yes | Snapshot, Cloning |
Portworx | pxd.openstorage.org | v0.3, v1.1 | A Container Storage Interface (CSI) Driver for Portworx | Persistent | Read/Write Multiple Pods | Yes | Snapshot, Expansion |
Pure Storage CSI | pure-csi | v1.0 | A Container Storage Interface (CSI) Driver for Pure Storage's Pure Service Orchestrator | Persistent | "Read/Write Single Pod" with FlashArray and "Read/Write Multiple Pods" with FlashBlade | Yes | Snapshot, Cloning |
QingCloud CSI | disk.csi.qingcloud.com | v1.1 | A Container Storage Interface (CSI) Driver for QingCloud Block Storage | Persistent | Read/Write Single Pod | Yes | Snapshot, Expansion, Cloning |
QingStor CSI | csi-neonsan | v0.3 | A Container Storage Interface (CSI) Driver for NeonSAN storage system | Persistent | Read/Write Single Pod | Yes | Snapshot |
Quobyte | quobyte-csi | v0.2 | A Container Storage Interface (CSI) Driver for Quobyte | Persistent | Read/Write Multiple Pods | Yes | |
ROBIN | robin | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for ROBIN | Persistent | Read/Write Multiple Pods | Yes | Raw Block, Snapshot, Cloning |
SmartX | csi-smtx-plugin | v1.0 | A Container Storage Interface (CSI) Driver for SmartX ZBS Storage | Persistent | Read/Write Multiple Pods | Yes | Snapshot, Expansion |
SandStone | csi-sandstone-plugin | v1.0 | A Container Storage Interface (CSI) Driver for SandStone USP | Persistent | Read/Write Single Pod | Yes | |
StorageOS | storageos | v0.3, v1.0 | A Container Storage Interface (CSI) Driver for StorageOS | Persistent | Read/Write Multiple Pods | Yes | |
XSKY-EBS | csi.block.xsky.com | v1.0 | A Container Storage Interface (CSI) Driver for XSKY Distributed Block Storage (X-EBS) | Persistent | Read/Write Single Pod | Yes | Raw Block, Snapshot, Expansion, Cloning |
XSKY-EUS | csi.fs.xsky.com | v1.0 | A Container Storage Interface (CSI) Driver for XSKY Distributed File Storage (X-EUS) | Persistent | Read/Write Multiple Pods | Yes | |
Vault | secrets.csi.kubevault.com | v1.0 | A Container Storage Interface (CSI) Driver for mounting HashiCorp Vault secrets as volumes. | Ephemeral | N/A | N/A | |
vSphere | vsphere.csi.vmware.com | v1.0 | A Container Storage Interface (CSI) Driver for VMware vSphere | Persistent | Read/Write Single Pod | Yes | Raw Block |
YanRongYun | ? | v1.0 | A Container Storage Interface (CSI) Driver for YanRong YRCloudFile Storage | Persistent | Read/Write Multiple Pods | Yes |
Sample Drivers
Name | Status | More Information |
---|---|---|
Flexvolume | Sample | |
HostPath | v1.2.0 | Only use for a single node tests. See the Example page for Kubernetes-specific instructions. |
ImagePopulator | Prototype | Driver that lets you use a container image as an ephemeral volume. |
In-memory Sample Mock Driver | v0.3.0 | The sample mock driver used for csi-sanity |
NFS | Sample | |
Synology NAS | v1.0.0 | An unofficial (and unsupported) Container Storage Interface Driver for Synology NAS. |
VFS Driver | Released | A CSI plugin that provides a virtual file system. |
Troubleshooting
Known Issues
- [minikube-3378]: Volume mount causes minikube VM to become corrupted
Common Errors
Node plugin pod does not start with RunContainerError status
kubectl describe pod your-nodeplugin-pod
shows:
failed to start container "your-driver": Error response from daemon:
linux mounts: Path /var/lib/kubelet/pods is mounted on / but it is not a shared mount
Your Docker host is not configured to allow shared mounts. Take a look at this page for instructions to enable them.
External attacher can't find VolumeAttachments
If you have a Kubernetes 1.9 cluster, not being able to list VolumeAttachment
and the following error are due to the lack of the
storage.k8s.io/v1alpha1=true
runtime configuration:
$ kubectl logs csi-pod external-attacher
...
I0306 16:34:50.976069 1 reflector.go:240] Listing and watching *v1alpha1.VolumeAttachment from github.com/kubernetes-csi/external-attacher/vendor/k8s.io/client-go/informers/factory.go:86
E0306 16:34:50.992034 1 reflector.go:205] github.com/kubernetes-csi/external-attacher/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1alpha1.VolumeAttachment: the server could not find the requested resource
...
Please see the Kubernetes 1.9 page.
Problems with the external components
The external components images are under active development. It can happen that they become incompatible with each other. If the issues above above have been ruled out, contact the sig-storage team and/or run the e2e test:
go run hack/e2e.go -- --provider=local --test --test_args="--ginkgo.focus=Feature:CSI"