Familiarize yourself with K8ssandra utilities and procedures for monitoring, repair, and upgrading your Apache Cassandra® database!
Completion time: 10 minutes.
Important
You must complete the steps in Quick start before continuing.
In this quick start we’ll cover the following topics:
- Accessing nodetool commands like status, ring, and info.
- Configure port forwarding for the Prometheus and Grafana monitoring utilties as well as Reaper for Apache Cassandra® (Reaper).
- Accessing the K8ssandra monitoring utilities, Prometheus and Grafana.
- Accessing Reaper, an easy to use repair interface.
- Upgrading a K8ssandra cluster: to ensure you’re using the latest K8ssandra software, or to apply new settings.
Access the Apache Cassandra® nodetool utility
Cassandra’s nodetool utility is commonly used for a variety of monitoring and management tasks. You’ll need to run nodetool on your K8ssandra cluster using the kubectl exec
command, since there’s no external stand alone option available.
To run nodetool commands:
- Get a list of the running K8ssandra pods using
kubectl get
:kubectl get pods
Output:
NAME READY STATUS RESTARTS AGE k8ssandra-cass-operator-6666588dc5-qpvzg 1/1 Running 3 45h k8ssandra-dc1-default-sts-0 2/2 Running 0 115m k8ssandra-dc1-stargate-6f7f5d6fd6-sblt8 1/1 Running 12 45h k8ssandra-grafana-6c4f6577d8-fxfsd 2/2 Running 6 45h k8ssandra-kube-prometheus-operator-5556885bd6-st4fp 1/1 Running 3 45h k8ssandra-reaper-k8ssandra-5b6cc959b7-zzlzr 1/1 Running 15 45h k8ssandra-reaper-k8ssandra-schema-47qzk 0/1 Completed 0 45h k8ssandra-reaper-operator-cc46fd5f4-85mk5 1/1 Running 4 45h prometheus-k8ssandra-kube-prometheus-prometheus-0 2/2 Running 7 45h
The K8ssandra pod running Cassandra takes the form <k8ssandra-cluster-name>-<datacenter-name>-default-sts-<n>
and, in the example above is k8ssandra-dc1-default-sts-0
which we’ll use throughout the following sections.
Tip
Although not applicable to this quick start, additional K8ssandra Cassandra nodes will increment the final <n>
but the rest of the name will remain the same.
2. Run nodetool status
, using the Cassandra node name k8ssandra-dc1-default-sts-0
, and replacing <k8ssandra-username>
and <k8ssandra-password>
with the values you retrieved in Retrieve K8ssandra superuser credentials:kubectl exec -it k8ssandra-dc1-default-sts-0 -c cassandra -- nodetool -u <k8ssandra-username> -pw <k8ssandra-password> status
Output:
Datacenter: dc1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Owns Host ID Token Rack UN 10.244.1.12 215.3 KiB ? 75e52e51-edc9-49f8-84f6-f044999ac130 -1080085985719557225 default Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Tip
All nodes should have the status UN
or “Up Normal.”
Other useful nodetool commands include:
nodetool ring
which outputs all the tokens in the node:kubectl exec -it k8ssandra-dc1-default-sts-0 -c cassandra -- nodetool -u <k8ssandra-username> -pw <k8ssandra-password> ring
Output:Datacenter: dc1 ========== Address Rack Status State Load Owns Token 9126546575375666475 172.17.0.13 default Up Normal 597.42 KiB ? -9138166261715795932 172.17.0.13 default Up Normal 597.42 KiB ? -9120920057340937901 172.17.0.13 default Up Normal 597.42 KiB ? -9117737800555727340 172.17.0.13 default Up Normal 597.42 KiB ? -9058127181143818684 172.17.0.13 default Up Normal 597.42 KiB ? -8998548020695455271 ...
nodetool info
which provides load and uptime information:kubectl exec -it k8ssandra-dc1-default-sts-0 -c cassandra -- nodetool -u <k8ssandra-username> -pw <k8ssandra-password> ring
Output:ID : dec6a537-f00c-458a-bbc0-26b173675cc7 Gossip active : true Thrift active : true Native Transport active: true Load : 597.42 KiB Generation No : 1614265335 Uptime (seconds) : 9232 Heap Memory (MB) : 567.72 / 1024.00 Off Heap Memory (MB) : 0.00 Data Center : dc1 Rack : default Exceptions : 0 Key Cache : entries 39, size 3.46 KiB, capacity 51 MiB, 199 hits, 240 requests, 0.829 recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 25 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Chunk Cache : entries 6, size 384 KiB, capacity 224 MiB, 111 misses, 3472 requests, 0.968 recent hit rate, NaN microseconds miss latency Percent Repaired : 100.0% Token : (invoke with -T/--tokens to see all 256 tokens)
For details on all nodetool commands, see The nodetool utility.
Configure port forwarding
In order to access Cassandra utilities outside of the K8s cluster, if you don’t have an Ingress setup as described in Configure Ingress, you’ll need to configure port forwarding.
Begin by getting a list of your K8ssandra K8s services and ports:
kubectl get services
Output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cass-operator-metrics ClusterIP 10.99.98.218 <none> 8383/TCP,8686/TCP 21h k8ssandra-dc1-all-pods-service ClusterIP None <none> 9042/TCP,8080/TCP,9103/TCP 21h k8ssandra-dc1-service ClusterIP None <none> 9042/TCP,9142/TCP,8080/TCP,9103/TCP,9160/TCP 21h k8ssandra-dc1-stargate-service ClusterIP 10.106.70.148 <none> 8080/TCP,8081/TCP,8082/TCP,8084/TCP,8085/TCP,9042/TCP 21h k8ssandra-grafana ClusterIP 10.96.120.157 <none> 80/TCP 21h k8ssandra-kube-prometheus-operator ClusterIP 10.97.21.175 <none> 443/TCP 21h k8ssandra-kube-prometheus-prometheus ClusterIP 10.111.184.111 <none> 9090/TCP 21h k8ssandra-reaper-k8ssandra-reaper-service ClusterIP 10.104.46.103 <none> 8080/TCP 21h k8ssandra-seed-service ClusterIP None <none> <none> 21h kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21h prometheus-operated ClusterIP None <none> 9090/TCP 2
In the output above, the services of interest are:
- k8ssandra-grafana: The K8ssandra grafana service where the name is a combination of the K8ssandra cluster name you specified during the Helm install,
k8ssandra
, and the postfix,-grafana
. This service listens on the internal K8s port80
. - prometheus-operated: The K8ssandra Prometheus daemon. This service listens on the internal K8s port
9090
. - k8ssandra-reaper-k8ssandra-reaper-service: The K8ssandra Reaper service where the name is a combination of the K8ssandra cluster name you specified during the Helm install,
k8ssandra
,-reaper
, the K8ssandra cluster name again, and the postfix-reaper-service
. This port listens on the internal K8s port8080
.
To configure port forwarding:
- Open a new terminal.
- Run the following 3
kubectl port-forward
commands in the background:kubectl port-forward svc/k8ssandra-grafana 9191:80 & kubectl port-forward svc/prometheus-operated 9292:9090 & kubectl port-forward svc/k8ssandra-reaper-k8ssandra-reaper-service 9393:8080 &
Output:[1] 29211 [2] 29212 [3] 29213 ~/ Forwarding from 127.0.0.1:9292 -> 9090 Forwarding from [::1]:9292 -> 9090 Forwarding from 127.0.0.1:9393 -> 8080 Forwarding from [::1]:9393 -> 8080 Forwarding from 127.0.0.1:9191 -> 3000 Forwarding from [::1]:9191 -> 3000
The K8ssandra services are now available at:
- Prometheus: http://127.0.0.1:9292
- Grafana: http://127.0.0.1:9191
- Reaper: http://127.0.0.1:9393/webui
Terminate port forwarding
To terminate a particular forwarded port:
- Get the process ID:
jobs -l
Output:[3] + 29213 running kubectl port-forward svc/k8ssandra-reaper-k8ssandra-reaper-service 9393:8080
- Kill the process
kill 80940
Output:[3] + terminated kubectl port-forward svc/k8ssandra-reaper-k8ssandra-reaper-service 9393:8080
Tip
Exiting the terminal instance will terminate all port forwarding services.
Access K8ssandra monitoring utilities
K8ssandra includes the following customized monitoring utilties:
- Prometheus a standard metrics collection and alerting tool.
- Grafana a set of pre-configured dashboards displaying important K8ssandra metrics.
Prometheus
To check on the health of your K8ssandra cluster using the K8ssandra Prometheus interface:
- Access the Prometheus home page at http://127.0.0.1:9292:
2. From the Status menu, choose Targets.
3. Verify that the stargate/0
and k8ssandra/0
are in the state UP
:
For more details on Prometheus, see the Prometheus web site.
Grafana
To monitor the health and performance of your K8ssandra cluster using the pre-configured K8ssandra dashboards:
- Retrieve the Grafana login username using the
helm show
command:helm show values k8ssandra/k8ssandra | grep "adminUser"
Output:admin
- Retrieve the Grafana login password using the
helm show
command:helm show values k8ssandra/k8ssandra | grep "adminPassword"
Output:secret
- Access the Grafana login screen at http://127.0.0.1:9191 and login using the username and password:
4. Click the home button indicated by the arrow:
5. Click the K8ssandra Overview
dashboard:
6. The K8ssandra Overview
dashboard is displayed:
For more information see the Grafana web site.
Access Reaper (Reaper)
The Reaper an easy interface for managing K8ssandra cluster repairs.
For more details, see the Reaper web site.
Upgrade K8ssandra
You can easily upgrade your K8ssandra software with the helm repo update
command, or apply new settings with the helm upgrade
command. For details, see Upgrade K8ssandra.
Next
See the K8ssandra task-based topics.