What you will learn
A RUNNING process may be waiting, stalled, or pointed at an idle trail. This tutorial shows a manual baseline that can later be implemented with a supported monitoring integration.
Before you begin
Use an isolated Oracle Database/GoldenGate combination certified for your exact platform and release. The examples use GoldenGate 19c Classic Architecture and GGSCI; Microservices deployments use different administrative workflows. Configure credential-store aliases, required database privileges, keys, supplemental logging, and a verified backup. Example groups are EDEMO (Extract), PDEMO (pump), and RDEMO (Replicat); first confirm the actual names in your lab.
1. Capture the process inventory
INFO ALL
INFO EXTRACT EDEMO, DETAIL
INFO REPLICAT RDEMO, DETAIL
Save process type, trail prefix, sequence/RBA, lag, checkpoint age, and database/service identity. A process can be RUNNING while no useful transaction is reaching the target.
2. Measure lag and progress
LAG EXTRACT EDEMO
LAG REPLICAT RDEMO
STATS REPLICAT RDEMO, TOTAL
Compare repeated observations during a known source transaction. Distinguish current lag from lag recorded at the last checkpoint. On an idle source, an old checkpoint alone is not proof of an outage.
3. Inspect reports and storage
VIEW REPORT EDEMO
VIEW REPORT RDEMO
Review ggserr.log, discard files where configured, trail volume, filesystem capacity, and archive retention. Keep credential-bearing parameter output out of shared logs. A successful process restart should not erase the error evidence.
4. Define actionable alerts
Use separate conditions for ABENDED/stopped unexpectedly, rising end-to-end lag, no progress during known traffic, and low trail/archive space. Set duration and thresholds from the service’s freshness budget. Route each alert with group, source/target identity, first-failure time, and the relevant report excerpt.
5. Test the monitoring path
In the lab, pause apply, generate a small transaction, and confirm the freshness alert triggers. Resume apply and verify both catch-up and alert recovery. Where supported, configure the official heartbeat mechanism to measure the complete path.
Locate a stalled component with a committed probe
Use a mapped lab table and a fresh synthetic key. Commit one source row, then inspect each component on its actual host. On host1.sample.com run:
INFO EXTRACT EDEMO, DETAIL
INFO EXTRACT PDEMO, DETAIL
STATS EXTRACT EDEMO, TOTAL
STATS EXTRACT PDEMO, TOTAL
On host2.sample.com run:
INFO REPLICAT RDEMO, DETAIL
STATS REPLICAT RDEMO, TOTAL
VIEW REPORT RDEMO
Confirm the exact probe row through SQL on the target. Source redo progress without local-trail progress points toward capture; local-trail progress without remote-trail progress points toward transport; an advancing remote trail with stale target data points toward apply or mapping. These are investigation directions, not conclusions from a single counter.
Do not compare absolute RBAs across different trail files as if they were the same position. Read sequence and offset together and track progress within each process. Statistics can reset after restart, so preserve timestamps and process start times. Monitor disk consumption against the longest planned outage and the slowest consumer before approving trail purges.
In an idle system, repeat with a committed probe rather than treating a quiet checkpoint as a guaranteed fault. In a busy system, row counts and zero displayed lag still need payload verification. Keep alerts tied to the service freshness objective and record successful alert recovery after the target catches up.
Example output
The following is an illustrative, normalized lab result, not output captured from a live customer system. Your versions, addresses, timings, and row counts will differ.
Illustrative INFO ALL snapshot:
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EDEMO 00:00:00 00:00:02
EXTRACT RUNNING PDEMO 00:00:00 00:00:01
REPLICAT RUNNING RDEMO 00:00:00 00:00:01
This normalized snapshot shows the example groups only; real group placement differs by host. A pump is reported as EXTRACT. RUNNING and zero checkpoint lag are not proof of data correctness—commit a synthetic transaction and verify its target result as described above.
Verify the result
A deliberate pause produces a visible, correctly routed alert and resuming apply clears it only after the target catches up. Preserve an example of each alert with its investigation steps.
Troubleshooting
Repeated alerts on idle tables may mean checkpoint age is being confused with active lag. Healthy process status with stale data calls for checking mappings and trail progression. A full trail filesystem requires investigating the lagging consumer before purging anything.
Recovery and next steps
Keep monitoring read-only. Automatic restarts or purges need separate controls because they can hide root causes or remove required recovery data. After an incident, retain reports, checkpoint evidence, and the final catch-up transaction.