What you will learn
Understand the difference between changing replica placement, streaming existing data, and changing client routing. These are separate operations with separate acceptance checks.
Before you begin
Use a disposable Cassandra 4.1 lab with three nodes in dc1, distinct node addresses, and a dedicated CQL account. The keyspace is tutorial; its replication factor is 3. Run nodetool on a database node with the deployment’s JMX authentication configured. Record the schema and confirm a restorable backup before changing topology. Hostnames ending in .sample.com identify roles; replace them with your own lab addresses.
1. Make the surviving datacenter independently usable
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.Confirm the surviving datacenter already has the required replicas and capacity. Repair and validate before reducing placement; the remaining copies must be the recovery source.
2. Move clients and background jobs
Change driver local-datacenter settings and contact points, move scheduled jobs, then observe traffic. Do not assume changing DNS migrates long-lived connection pools.
3. Remove the old datacenter from each relevant keyspace
ALTER KEYSPACE tutorial WITH replication = {'class':'NetworkTopologyStrategy','dc1':3};
This example retains dc1 and retires dc2. Verify the actual datacenter names and all application keyspaces. Audit system_auth placement separately. Wait for schema agreement and validate reads/writes in dc1.
4. Retire membership in a controlled sequence
Use the version-supported datacenter removal procedure, with live-node decommission where applicable. Complete and verify each membership change before the next. Forced removal and assassination are not normal shortcuts for healthy nodes. Retain the old data and backup until acceptance.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative topology acceptance record:
Remaining datacenter: dc1
Remaining intended nodes: 3
Node states: UN, UN, UN
Unexpected endpoints: 0
Application read/write test: passed
This is a compact acceptance record assembled from the membership and application checks, not verbatim nodetool output. UN means Up/Normal. A DN, UJ, or UL state requires investigation before proceeding.
Verify the result
Compare the final ring and schema with the written target. Read and write known partitions through the surviving service path at the application’s real consistency level. Confirm no unexpected unreachable endpoints remain.
Troubleshooting
If streaming stops progressing, check peer reachability, disk space, pending compaction, and source replica availability. If queries fail after a client move, inspect local-datacenter and consistency settings before blaming data loss.
Recovery and next steps
Before replica-map changes, move traffic back only if the old datacenter is healthy. After removal, restoring its name in a replication map is not a data recovery procedure: replicas may require rebuild/repair. Preserve a recoverable source and stop if the surviving datacenter cannot meet consistency requirements.