Degraded API Latency

Incident Report for Sardine AI

Postmortem

Summary

Our cache service experienced intermittent latency issues on May 22, 2026. Service has been fully restored.

What Happened

Our cache infrastructure experienced three distinct periods of elevated latency:

  • 22:23-22:31 UTC
  • 22:46-22:52 UTC
  • 23:00-23:05 UTC

The system partially recovered between incidents but experienced cascading failures before full restoration at 23:05 UTC

Why It Happened

The incident began with an unusual spike in cache sets and cache deletes that stressed the caching infrastructure.

What We're Doing About It

  • Implementing infrastructure improvements to prevent similar incidents resulting from noisy neighbor pattern
  • Enhancing cache monitoring, alerting and run books
  • Working with SME’s from our cloud provider to identify and address any contributing factors

We apologize for the disruption and appreciate your patience.

Posted May 26, 2026 - 22:57 UTC

Resolved

From 22:22 UTC to to 23:05 UTC elevated latency could have affected some API requests.
Posted May 22, 2026 - 22:00 UTC