What you will learn
Understand the difference between changing replica placement, streaming existing data, and changing client routing. These are separate operations with separate acceptance checks.
Before you begin
Use a disposable Cassandra 4.1 lab with three nodes in dc1, distinct node addresses, and a dedicated CQL account. The keyspace is tutorial; its replication factor is 3. Run nodetool on a database node with the deployment’s JMX authentication configured. Record the schema and confirm a restorable backup before changing topology. Hostnames ending in .sample.com identify roles; replace them with your own lab addresses.
1. Baseline the existing ring
nodetool status
nodetool describecluster
nodetool netstats
The expected starting state is every intended node Up/Normal (UN), one schema version, and no unfinished topology operation. A joining or leaving node is a reason to investigate before starting another change. Also record read/write latency and disk headroom; ring membership alone does not demonstrate application health.
2. Join the new datacenter
Provision three nodes with dc=dc2, the existing cluster name, compatible partitioner/snitch settings, and distinct addresses. Follow the release-specific new-datacenter joining procedure; keep clients local to dc1 while dc2 is empty. Do not let an empty new datacenter satisfy application reads.
3. Extend replica placement
ALTER KEYSPACE tutorial WITH replication = {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};
Audit every application keyspace separately. Changing this map does not itself prove existing data has arrived. Review authentication-keyspace replication separately under your security procedure.
4. Populate and exercise the new replicas
nodetool rebuild dc1
nodetool netstats
Run rebuild on each new node according to the topology procedure, allowing streaming to finish. Test representative partitions at local quorum from a client configured for dc2; compare with dc1. Enable traffic gradually only after validation.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative topology acceptance record:
dc1: 3 intended nodes, all UN
dc2: 3 intended nodes, all UN
Pending streaming: none
Read/write test in dc2: passed
This is a compact acceptance record assembled from the membership and application checks, not verbatim nodetool output. UN means Up/Normal. A DN, UJ, or UL state requires investigation before proceeding.
Verify the result
Compare the final ring and schema with the written target. Read and write known partitions through the surviving service path at the application’s real consistency level. Confirm no unexpected unreachable endpoints remain.
Troubleshooting
If streaming stops progressing, check peer reachability, disk space, pending compaction, and source replica availability. If queries fail after a client move, inspect local-datacenter and consistency settings before blaming data loss.
Recovery and next steps
Before routing traffic to dc2, the existing datacenter remains the service path. If streaming fails, keep clients on dc1, diagnose unavailable source replicas or network limits, and resume using the supported procedure. Do not clean up source data until the topology and replica checks are complete.