debug-production-incident

Prompt

Our user dashboard is showing 500 errors for all users in the US East region since 10:15 AM. The error rate spiked suddenly after a recent deployment. Can you help?

Output

SITUATION: US East users experiencing 500 errors since 10:15 AM, coinciding with recent deployment, indicating a likely deploy-related incident.
TIMELINE: Deployment completed at 10:10 AM; errors began at 10:15 AM; alert triggered at 10:16 AM.
HYPOTHESES: Deploy introduced a breaking change; logs show new error signatures post-deploy; metrics indicate error spike immediately after release.
ACTION NOW: Roll back the recent deployment to the previous stable version. Expect error rate to decrease and errors to subside within 2-3 minutes.
ROLLBACK: Revert the deployment via the deployment system to the last known-good build. Monitor metrics to confirm recovery.

Prompt

Our API service is experiencing high latency and increased error rates in the Europe region, affecting 30% of users. No recent deploys. What should I check next?

Output

SITUATION: Europe region API experiencing high latency and errors affecting 30% of users, without recent deployments, indicating a possible dependency or resource issue.
TIMELINE: Latency and errors increased sharply at 11:45 AM; no recent code changes; traffic volume stable.
HYPOTHESES: Dependency slowdown (e.g., downstream API or database); resource saturation (CPU/memory); network issues.
ACTION NOW: Check dependency health and latency metrics, focusing on downstream services and database performance. If dependency latency is high, enable circuit breakers or serve cached responses.
ROLLBACK: Not applicable unless a dependency fix is identified; focus on mitigating by serving degraded responses or rate limiting until root cause is confirmed.