What you will learn
PostgreSQL supplies replication and promotion primitives, not a complete automatic failover controller. This tutorial builds a decision and failure-test plan for your chosen HA manager; it does not present a standalone promotion script as an HA system.
Before you begin
Start with a validated streaming standby and a tested backup. Inventory all writers and connection pools. Establish a fencing method that prevents the old primary accepting writes, even during a network partition. Agree the acceptable data-loss and interruption budgets. Keep a console connection to both lab nodes.
1. Prove the candidate is eligible
SELECT pg_is_in_recovery();
SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();
Run on the standby. Check the primary’s pg_stat_replication, replay lag, and WAL availability. Wall-clock lag may be misleading on an idle system; compare positions and known committed transactions.
2. Control the old writer
Configure the HA manager’s quorum, leader lease and fencing integration. Test loss of the primary process, loss of the candidate, and a network partition separately. A ping failure alone must not authorize a second primary.
3. Promote through the chosen control path
Use the HA manager’s documented switchover/failover operation so its state and routing agree. Do not run manual pg_ctl promotion alongside an active manager. Record the election, fencing, promotion, and endpoint-change timestamps.
4. Route and reconnect clients
jdbc:postgresql://host1.sample.com:5432,host2.sample.com:5432/tutorial?targetServerType=primary
A multi-host JDBC URL can select a writable server; it does not perform leader election or prevent split brain. Refresh pools and verify the connection actually reaches the new primary. Include retries of ambiguous commit outcomes in the application test.
5. Rejoin the old server as a standby
Keep the old writer fenced. Check pg_rewind prerequisites and required WAL before using it, or build a fresh standby with pg_basebackup. Never restart the old data directory as a writable primary while the new primary is serving.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
-- On the promoted node
SELECT pg_is_in_recovery();
pg_is_in_recovery
-------------------
f
(1 row)
-- Acceptance record outside SQL
Old primary fenced: confirmed
Writable database endpoints: 1
False means this node is no longer in recovery. It does not prove that the old primary is fenced: verify that independently before allowing writes. The last two lines are operator checks, not PostgreSQL output.
Verify the result
Exactly one server accepts writes. A test transaction reaches the current primary and appears on the rejoined standby. Measure interruption and verify that routing, monitoring, backup jobs, and HA-controller state identify the same leader.
Troubleshooting
Two writable nodes mean fencing failed: stop client writes and isolate the conflict before reconciliation. A promoted node with no application traffic suggests stale routing or connection pools. A rejoin failure can reflect missing WAL or unmet rewind prerequisites.
Recovery and next steps
After promotion and new writes, returning to the old server is another controlled transition, not an undo button. Rebuild/rewind it as a standby, wait for synchronization, then rehearse a fresh switchover. Retain logs and the transaction boundary for reconciliation.