What you will learn
The original version sequence is a historical incident scenario. Build a fresh isolated copy and choose eligible source/target patch releases from the release-specific upgrade notes; do not treat 3.11.8, 3.11.17, or 4.1.4 as current recommendations. Reproduce the client workload that previously lost availability.
Before you begin
Use a disposable Cassandra 4.1 lab with three nodes in dc1, distinct node addresses, and a dedicated CQL account. The keyspace is tutorial; its replication factor is 3. Run nodetool on a database node with the deployment’s JMX authentication configured. Record the schema and confirm a restorable backup before changing topology. Hostnames ending in .sample.com identify roles; replace them with your own lab addresses. For cross-version work, consult the target release’s NEWS file and Java/driver support details. Snapshot data and schema, record configuration, and prove the restore path in the lab.
1. Record the baseline
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.
nodetool version
nodetool tablestats tutorial.events
Keep application success rates and tail latency beside these outputs.
2. Prepare the compatibility boundary
List the exact old and new builds, JVMs, and changed settings. Preserve the previous software home and configuration. Stop scheduled repair/topology operations that would interfere with the maintenance work. Do not attempt simultaneous upgrades across replica peers.
3. Perform the canary operation
nodetool drain
Drain the selected node, then stop its configured service cleanly. Apply the approved binary/JVM/configuration change and start that service. Drain flushes work; it is not a backup. Review the first startup log for rejected options, schema errors, and storage-format problems.
4. Exercise clients and decide whether to continue
Wait for normal membership and successful quorum reads/writes. Confirm the driver reconnects and requests do not accumulate at the coordinator. Compare metrics with the baseline. If this canary fails, stop the rollout; do not upgrade additional nodes to make the versions look consistent.
5. Complete the release transition
Repeat the validated operation one node at a time, using the release’s mixed-version restrictions. After all binaries are accepted, plan SSTable conversion if required. Resume background maintenance only after the cluster is stable and record the resulting version inventory.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative canary comparison:
Checkpoint Before After
Intended node state UN UN
Running build source build approved target build
Application write succeeds succeeds
Application read correct value correct value
Unreviewed startup errors 0 0
Fill this comparison with the actual nodetool version, status, log, and client-test results. Do not mark the build check successful merely because the service process started. SSTable conversion has its own disk-space and I/O acceptance checks.
Verify the result
Every intended node reports the target build, expected membership, and healthy application traffic. The upgrade test records reconnect behavior, read/write failures, compaction pressure, and a successful restore rehearsal rather than only the absence of fatal log messages.
Troubleshooting
If a node fails during startup, compare the target release’s configuration names and Java requirements. If all clients fail when one node stops, inspect contact points, local datacenter, driver retries, and actual replica placement. For SSTable conversion pressure, review disk headroom and compaction throughput.
Recovery and next steps
Do not assume reinstalling old binaries is safe after new-format files or schema changes exist. Use only the downgrade path supported by the exact release pair, or restore the verified pre-upgrade backup into a compatible environment. Keep the application recovery decision separate from the binary rollback decision.