The K8ssandra team recently released K8ssandra-operator v1.1. Even though K8ssandra is still evolving, a large number of features have been added ranging from multi-cluster support to restoring a Medusa backup from any Apache Cassandra® cluster. With all of these capabilities, we need to ensure quality checks are executed at various levels from unit to cloud verification.
We added an upfront test focus in the construction of K8ssandra-operator v1.0, with numerous test scenarios primarily scoped to unit, integration, and e2e. Currently, these tests can run locally or as part of the CI/CD (continuous integration and continuous delivery) process.
Automated cloud smoke tests were becoming increasingly important as we built K8ssandra features. Specifically, to ensure cloud-specific issues are discovered early in the development process. So we built a K8ssandra cloud-readiness framework to support the development of “in cloud” tests for new versions of K8ssandra. Here’s a quick look at how this all works.
Architecture and technologies
In a nutshell, the cloud-readiness framework is a GoLang module that plugs-in reusable cloud infrastructure provisioning, creates Kubernetes resources, and optionally deploys a K8ssandra ecosystem in a single or multi-cluster configuration.
Version 0.1 for cloud-readiness leverages Terraform modules for infrastructure provisioning as this was something the K8ssandra team already had available. The technologies we used for this solution include the following:
A conscious decision was made to not only reuse existing provisioning, but also include simulation and verbose logging. This combination makes troubleshooting and test failure diagnostics easier given optional settings available for Kubernetes and cloud deployments.
For example, to enable a simulation for provisioning, it is as simple as assigning the value true to the “Simulate” flag and running your specific scenario using the command: go test -v <your-test>.
Readiness models
Cloud-readiness models are what inform the test scenario to provision and install for each cloud environment.
As you construct scenario models, you will notice that K8ssandra tests (i.e. smoke) reference their own reusable named packages. The idea is to allow the reuse of test scenario definitions involving cloud-specific setup and resource creation.
Currently, there is a cloud context model and readiness model used to separate areas of concern, such as: cloud resource/infrastructure setup, K8ssandra installation, and pre-installation activities.
Testing the tests
At the top level of the cloud-readiness project, there is another test folder. This folder is not specific to K8ssandra verifications, rather it is used to verify the overall automation provided by the framework.
This enables “testing the tester,” so to speak. It can verify the framework components to ensure future changes don’t break existing and new scenarios.
Kicking the tires
A cloud-readiness user guide for v0.1 is available and walks you through how to get started with your initial scenario construction. What’s more, all of this is part of the K8ssandra open-source projects.
The decoupled modules and components allow for mixing and matching your own set of test frameworks, APIs, deployments, etc. For example, if you want to use Pulumi instead of Terraform, just fork the cloud-readiness repository and customize your Go-based test pre-conditions to reference those bits. Then, only reference the installation activities for K8ssandra (pre-install Cert-manager and K8ssandra ecosystem install).
Also, if you would like to run a set of system tests or smoke tests, you can create a custom GoLang test to plugin infrastructure provisioning, the deployment of K8ssandra, and the deployment of your own Kubernetes resources.
Next steps
The cloud-readiness is still in its infancy with v0.1, so stay tuned for more updates as we roll out additional features. As always, feel free to open enhancement requests, fork the repository and contribute—we love to hear from you!
Join the K8ssandra community on Twitter or on GitHub to hear our latest releases and upgrades. Browse the K8ssandra blog for more tutorials on K8ssandra features.