Blog: Testing of CSI drivers
-
Author: Patrick Ohly (Intel)
When developing a Container Storage Interface (CSI) driver, it is useful to leverage as much prior work as possible. This includes source code (like the sample CSI hostpath driver) but also existing tests. Besides saving time, using tests written by someone else has the advantage that it can point out aspects of the specification that might have been overlooked otherwise.
An earlier blog post about end-to-end testing already showed how to use the Kubernetes storage tests for testing of a third-party CSI driver. That approach makes sense when the goal to also add custom E2E tests, but depends on quite a bit of effort for setting up and maintaining a test suite.
When the goal is to merely run the existing tests, then there are simpler approaches. This blog post introduces those.
Sanity testing
csi-test sanity ensures that a CSI driver conforms to the CSI specification by calling the gRPC methods in various ways and checking that the outcome is as required. Despite its current hosting under the Kubernetes-CSI organization, it is completely independent of Kubernetes. Tests connect to a running CSI driver through its Unix domain socket, so although the tests are written in Go, the driver itself can be implemented in any language.
The main README explains how to include those tests into an existing Go test suite. The simpler alternative is to just invoke the csi-sanity command.
Installation
Starting with csi-test v3.0.0, you can build the
csi-sanity
command withgo get github.com/kubernetes-csi/csi-test/cmd/csi-sanity
and you’ll find the compiled binary in$GOPATH/bin/csi-sanity
.go get
always builds the latest revision from the master branch. To build a certain release, get the source code and runmake -C cmd/csi-sanity
. This producescmd/csi-sanity/csi-sanity
.Usage
The
csi-sanity
binary is a full Ginkgo test suite and thus has the usual-gingko
command line flags. In particular,-ginkgo.focus
and-ginkgo.skip
can be used to select which tests are run resp. not run.During a test run,
csi-sanity
simulates the behavior of a container orchestrator (CO) by creating staging and target directories as required by the CSI spec and calling a CSI driver via gRPC. The driver must be started before invokingcsi-sanity
. Although the tests currently only check the gRPC return codes, that might change and so the driver really should make the changes requested by a call, like mounting a filesystem. That may mean that it has to run as root.At least one gRPC endpoint must be specified via the
-csi.endpoint
parameter when invokingcsi-sanity
, either as absolute path (unix:/tmp/csi.sock
) for a Unix domain socket or as host name plus port (dns:///my-machine:9000
) for TCP.csi-sanity
then uses that endpoint for both node and controller operations. A separate endpoint for controller operations can be specified with-csi.controllerendpoint
. Directories are created in/tmp
by default. This can be changed via-csi.mountdir
and-csi.stagingdir
.Some drivers cannot be deployed such that everything is guaranteed to run on the same host. In such a case, custom scripts have to be used to handle directories: they log into the host where the CSI node controller runs and create or remove the directories there.
For example, during CI testing the CSI hostpath example driver gets deployed on a real Kubernetes cluster before invoking
csi-sanity
and thencsi-sanity
connects to it through port forwarding provided bysocat
. Scripts are used to create and remove the directories.Here’s how one can replicate that, using the v1.2.0 release of the CSI hostpath driver:
$ cd csi-driver-host-path $ git describe --tags HEAD v1.2.0 $ kubectl get nodes NAME STATUS ROLES AGE VERSION 127.0.0.1 Ready <none> 42m v1.16.0 $ deploy/kubernetes-1.16/deploy-hostpath.sh applying RBAC rules kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.4.0/deploy/kubernetes/rbac.yaml ... deploying hostpath components deploy/kubernetes-1.16/hostpath/csi-hostpath-attacher.yaml using image: quay.io/k8scsi/csi-attacher:v2.0.0 service/csi-hostpath-attacher created statefulset.apps/csi-hostpath-attacher created deploy/kubernetes-1.16/hostpath/csi-hostpath-driverinfo.yaml csidriver.storage.k8s.io/hostpath.csi.k8s.io created deploy/kubernetes-1.16/hostpath/csi-hostpath-plugin.yaml using image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0 using image: quay.io/k8scsi/hostpathplugin:v1.2.0 using image: quay.io/k8scsi/livenessprobe:v1.1.0 ... service/hostpath-service created statefulset.apps/csi-hostpath-socat created 07:38:46 waiting for hostpath deployment to complete, attempt #0 deploying snapshotclass volumesnapshotclass.snapshot.storage.k8s.io/csi-hostpath-snapclass created $ cat >mkdir_in_pod.sh <<EOF #!/bin/sh kubectl exec csi-hostpathplugin-0 -c hostpath -- mktemp -d /tmp/csi-sanity.XXXXXX EOF $ cat >rmdir_in_pod.sh <<EOF #!/bin/sh kubectl exec csi-hostpathplugin-0 -c hostpath -- rmdir "\$@" EOF $ chmod u+x *_in_pod.sh $ csi-sanity -ginkgo.v \ -csi.endpoint dns:///127.0.0.1:$(kubectl get "services/hostpath-service" -o "jsonpath={..nodePort}") \ -csi.createstagingpathcmd ./mkdir_in_pod.sh \ -csi.createmountpathcmd ./mkdir_in_pod.sh \ -csi.removestagingpathcmd ./rmdir_in_pod.sh \ -csi.removemountpathcmd ./rmdir_in_pod.sh Running Suite: CSI Driver Test Suite ==================================== Random Seed: 1570540138 Will run 72 of 72 specs ... Controller Service [Controller Server] ControllerGetCapabilities should return appropriate capabilities /nvme/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:111 STEP: connecting to CSI driver STEP: creating mount and staging directories STEP: checking successful response • ------------------------------ Controller Service [Controller Server] GetCapacity should return capacity (no optional values added) /nvme/gopath/src/github.com/kubernetes-csi/csi-test/pkg/sanity/controller.go:149 STEP: reusing connection to CSI driver at dns:///127.0.0.1:30056 STEP: creating mount and staging directories ... Ran 53 of 72 Specs in 148.206 seconds SUCCESS! -- 53 Passed | 0 Failed | 0 Pending | 19 Skipped PASS
Some comments:
- The source code of these tests is in the
pkg/sanity
package. - How to determine the external IP address of the node depends on the
cluster. In this example, the cluster was brought up with
hack/local-up-cluster.sh
and thus runs on the local host (127.0.0.1
). It uses a port allocated by Kubernetes, obtained above withkubectl get "services/hostpath-service"
. The Kubernetes-CSI CI uses kind and there a Docker command can be used. - The create script must print the final directory. Using a unique directory for each test case has the advantage that if something goes wrong in one test case, others still start with a clean slate.
- The “staging directory”, aka
NodePublishVolumeRequest.target_path
in the CSI spec, must be created and deleted by the CSI driver while the CO is responsible for the parent directory.csi-sanity
handles that by creating a directory and then giving the CSI driver that directory path with/target
appended at the end. Kubernetes got this wrong and creates the actualtarget_path
directory, so CSI drivers which want to work with Kubernetes currently have to be lenient and must not fail when that directory already exists. - The “mount directory” corresponds to
NodeStageVolumeRequest.staging_target_path
and really gets created by the CO, i.e.csi-sanity
.
End-to-end testing
In contrast to
csi-sanity
, end-to-end testing interacts with the CSI driver through the Kubernetes API, i.e. it simulates operations from a normal user, like creating a PersistentVolumeClaim. Support for testing external CSI drivers was added in Kubernetes 1.14.0.Installation
For each Kubernetes release, a test tar archive is published. It’s not listed in the release notes (for example, the ones for 1.16), so one has to know that the full URL is
https://dl.k8s.io/<version>/kubernetes-test-linux-amd64.tar.gz
(like for v1.16.0).These include a
e2e.test
binary for Linux on x86-64. Archives for other platforms are also available, see this KEP. Thee2e.test
binary is completely self-contained, so one can “install” it and theginkgo
test runner with:curl --location https://dl.k8s.io/v1.16.0/kubernetes-test-linux-amd64.tar.gz | \ tar --strip-components=3 -zxf - kubernetes/test/bin/e2e.test kubernetes/test/bin/ginkgo
Each
e2e.test
binary contains tests that match the features available in the corresponding release. In particular, the[Feature: xyz]
tags change between releases: they separate tests of alpha features from tests of non-alpha features. Also, the tests from an older release might rely on APIs that were removed in more recent Kubernetes releases. To avoid problems, it’s best to simply use thee2e.test
binary that matches the Kubernetes release that is used for testing.Usage
Not all features of a CSI driver can be discovered through the Kubernetes API. Therefore a configuration file in YAML or JSON format is needed which describes the driver that is to be tested. That file is used to populate the driverDefinition struct and the DriverInfo struct that is embedded inside it. For detailed usage instructions of individual fields refer to these structs.
A word of warning: tests are often only run when setting some fields and the file parser does not warn about unknown fields, so always check that the file really matches those structs.
Here is an example that tests the
csi-driver-host-path
:$ cat >test-driver.yaml <<EOF StorageClass: FromName: true SnapshotClass: FromName: true DriverInfo: Name: hostpath.csi.k8s.io Capabilities: block: true controllerExpansion: true exec: true multipods: true persistence: true pvcDataSource: true snapshotDataSource: true InlineVolumes: - Attributes: {} EOF
At a minimum, you need to define the storage class you want to use in the test, the name of your driver, and what capabilities you want to test. As with
csi-sanity
, the driver has to be running in the cluster before testing it. The actuale2e.test
invocation then enables tests for this driver with-storage.testdriver
and selects the storage tests for it with-ginkgo.focus
:$ ./e2e.test -ginkgo.v \ -ginkgo.focus='External.Storage' \ -storage.testdriver=test-driver.yaml Oct 8 17:17:42.230: INFO: The --provider flag is not set. Continuing as if --provider=skeleton had been used. I1008 17:17:42.230210 648569 e2e.go:92] Starting e2e run "90b9adb0-a3a2-435f-80e0-640742d56104" on Ginkgo node 1 Running Suite: Kubernetes e2e suite =================================== Random Seed: 1570547861 - Will randomize all specs Will run 163 of 5060 specs Oct 8 17:17:42.237: INFO: >>> kubeConfig: /var/run/kubernetes/admin.kubeconfig Oct 8 17:17:42.241: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable ... ------------------------------ SSSSSSSSSSSSSSSSSSSS ------------------------------ External Storage [Driver: hostpath.csi.k8s.io] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node /workspace/anago-v1.16.0-rc.2.1+2bd9643cee5b3b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:191 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] ...
You can use
ginkgo
to run some kinds of test in parallel. Alpha feature tests or those that by design have to be run sequentially then need to be run separately:$ ./ginkgo -p -v \ -focus='External.Storage' \ -skip='[Feature:|[Disruptive]|[Serial]' \ ./e2e.test \ -- \ -storage.testdriver=test-driver.yaml $ ./ginkgo -v \ -focus='External.Storage.*([Feature:|[Disruptive]|[Serial])' \ ./e2e.test \ -- \ -storage.testdriver=test-driver.yaml
Getting involved
Both the Kubernetes storage tests and the sanity tests are meant to be applicable to arbitrary CSI drivers. But perhaps tests are based on additional assumptions and your driver does not pass the testing although it complies with the CSI specification. If that happens then please file issues (links below).
These are open source projects which depend on the help of those using them, so once a problem has been acknowledged, a pull request addressing it will be highly welcome.
The same applies to writing new tests. The following searches in the issue trackers select issues that have been marked specifically as something that needs someone’s help: - csi-test - Kubernetes
Happy testing! May the issues it finds be few and easy to fix.
https://kubernetes.io/blog/2020/01/08/testing-of-csi-drivers/
- The source code of these tests is in the
© Lightnetics 2024