Impact

High latency for v1/customers and v1/issuing-risk endpoints, latency was observed to be around 3.5 - 4 secs during the outage

Timeline

Latency reported around 5:12PM EST

Service restored ~2hours later at 7:20 PM EST

Root Cause

• Scaling issue - Primary DB instance was at max capacity (>90%) although the number of connections were low.

Resolution

Scaled up Primary instance from 8 vCPU, 64 GB to 16 vCPU, 128 GB . After scaling the db server, CPU utilization drops and latency came down. Since then CPU usage on primary has been below 50%.