Incident Overview

• Duration: Approximately 1 hours, from 2023-12-01 08:01:00 to 2023-12-01 09:05:00 PST
• Services Affected: Customers API, Issuing API
• Impact: elevated latency

Root Cause Analysis

• Primary Issue: The degradation was due to an increase in the traffic by multiple jobs via multiple clients that coincidentally runs on the first day of the month
• Second Factor: While traffic was not significant, traffic resulted in unexpected query pattern that caused slow database queries
• Third Factor: This coincided with recent database configuration change sardine was internally rolling out for performance optimization. During rollout we didn’t provision enough read replica resources
• Technical Impact: The result was a performance degradation

Action items

1. Optimize database configuration including provisioning more replica, use appropriate resource configuration, and optimizing parameters
2. Optimize query and access pattern