What you will learn
Understand the difference between changing replica placement, streaming existing data, and changing client routing. These are separate operations with separate acceptance checks.
Before you begin
Use a disposable Cassandra 4.1 lab with three nodes in dc1, distinct node addresses, and a dedicated CQL account. The keyspace is tutorial; its replication factor is 3. Run nodetool on a database node with the deployment’s JMX authentication configured. Record the schema and confirm a restorable backup before changing topology. Hostnames ending in .sample.com identify roles; replace them with your own lab addresses.
1. Check capacity and replica placement
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.For RF=3, keep at least three suitable nodes in the datacenter after removal. In a three-node lab, add and validate a fourth node first; removing one of only three cannot retain three distinct local replicas.
2. Remove the client dependency
Remove the retiring node from explicit client contact-point lists where necessary. Confirm drivers can discover and reach remaining nodes. Pause competing topology tasks; keep health monitoring enabled.
3. Decommission the live node
nodetool decommission
Run this on the node being removed and monitor streaming from another terminal. Do not stop a healthy node first and substitute forced removal. removenode is a different workflow for a dead node.
4. Confirm ownership has moved
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.Check from surviving nodes, exercise the application, and record the final host-ID inventory before disabling the retired service.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative topology acceptance record:
Remaining datacenter: dc1
Remaining intended nodes: 3
Node states: UN, UN, UN
Unexpected endpoints: 0
Application read/write test: passed
This is a compact acceptance record assembled from the membership and application checks, not verbatim nodetool output. UN means Up/Normal. A DN, UJ, or UL state requires investigation before proceeding.
Verify the result
Compare the final ring and schema with the written target. Read and write known partitions through the surviving service path at the application’s real consistency level. Confirm no unexpected unreachable endpoints remain.
Troubleshooting
If streaming stops progressing, check peer reachability, disk space, pending compaction, and source replica availability. If queries fail after a client move, inspect local-datacenter and consistency settings before blaming data loss.
Recovery and next steps
If decommission stalls, preserve the node and inspect streaming and logs before any retry. Do not restart a successfully decommissioned data directory as a new member. Reintroduction is a deliberate fresh-node/bootstrap operation after retaining the needed recovery copy.