What you will learn
Understand the difference between changing replica placement, streaming existing data, and changing client routing. These are separate operations with separate acceptance checks.
Before you begin
Use a disposable Cassandra 4.1 lab with three nodes in dc1, distinct node addresses, and a dedicated CQL account. The keyspace is tutorial; its replication factor is 3. Run nodetool on a database node with the deployment’s JMX authentication configured. Record the schema and confirm a restorable backup before changing topology. Hostnames ending in .sample.com identify roles; replace them with your own lab addresses.
1. Build the topology evidence
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.Save host IDs, datacenter/rack labels, and the intended final node count. List every application keyspace and its replication map.
2. Test the surviving service path
Run the application with contact points and local-datacenter selection restricted to the survivor. Include authentication, reads, writes, scheduled jobs, and connection-pool refresh. Record error and latency thresholds before the test.
3. Reconcile operational dependencies
Search deployment configuration, backup schedules, monitoring targets, dashboards, allowlists, and batch jobs for the retiring endpoint set. Assign a team role to each change, and retain recovery and security monitoring during the maintenance window.
4. Make the decision measurable
Approve retirement only when remaining replicas are healthy, backup restoration has been rehearsed, client tests pass, and the surviving capacity stays within agreed limits. A failed gate postpones retirement. This tutorial prepares evidence; it deliberately does not issue removal commands.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative topology acceptance record:
Remaining datacenter: dc1
Remaining intended nodes: 3
Node states: UN, UN, UN
Unexpected endpoints: 0
Application read/write test: passed
This is a compact acceptance record assembled from the membership and application checks, not verbatim nodetool output. UN means Up/Normal. A DN, UJ, or UL state requires investigation before proceeding.
Verify the result
Compare the final ring and schema with the written target. Read and write known partitions through the surviving service path at the application’s real consistency level. Confirm no unexpected unreachable endpoints remain.
Troubleshooting
If streaming stops progressing, check peer reachability, disk space, pending compaction, and source replica availability. If queries fail after a client move, inspect local-datacenter and consistency settings before blaming data loss.
Recovery and next steps
If the rehearsal fails, restore the previous client configuration and investigate. Keep the retiring datacenter available until the evidence is complete. Use the separate retirement tutorial for the actual topology change.