Issue with production API endpoint returning 504 Gateway Timeout

Our public /api/v1/transactions endpoint is intermittently timing out under high load, despite no obvious code changes. We've ruled out database locks and have sufficient worker capacity. Could this be related to a downstream service timeout or an issue at the Nginx reverse proxy level? Any advice on how to isolate which layer is failing would be greatly appreciated — we need to get this fixed before it impacts more customers.

A 504 is almost never a code-level failure in the immediate sense; it's a structural signaling mechanism from a proxy layer (Nginx, ALB, Cloudflare, etc.) indicating that an upstream service failed to produce a response within the configured timeout budget. To diagnose this systematically, we need to isolate where exactly the pipeline is stalling by correlating timestamps across every hop.

First, identify the proxy producing the 504. If you're behind Nginx as a reverse proxy for a Gunicorn/Uvicorn app, check your proxy_read_timeout and proxy_connect_timeout directives — if those are set to 60s but your upstream worker is actually taking 90s to complete the business logic (perhaps because of a slow RDS query or an unoptimized O(N^2) operation on a growing dataset), Nginx will sever the connection and return 504 before the app completes. The fix there isn't always to increase the timeout — increasing it only masks a throughput problem that will eventually saturate all worker threads under load. You have to find why the upstream is slow.

Second, examine your application-server thread/worker pool utilization. If you have 10 Gunicorn workers and each request takes 2 seconds (maybe due to an unindexed SQL query or waiting on a third-party API with no timeout), you can only handle 5 requests per second before the queue backs up and subsequent connections time out at the proxy level. A p99 of 2s is fine; a p100 of 30s is lethal because it ties up workers that should be handling new traffic. You'll want to check psutil or your APM (Datadog, New Relic) for worker saturation.

Third — and this is the one most people miss in production hotfixes — set explicit timeouts on every outbound HTTP request from within

um... i'm so sorry to bother but does this mean the whole site is down or just one part? and what exactly does 504 mean -- i looked it up and there are like three different explanations and i don't know which applies here. also should we be panicking because my manager keeps asking for updates and i don't want to say anything wrong... sorry if this is a stupid question

Check the logs first. There's literally no context in this post — what endpoint, which region, what payload size? 504 could be anything from a downstream service hanging to your load balancer configuration being wrong. The issue tracker already has five open threads on this exact topic with extensive

what does gateway timeout even mean — is it our server or theirs? i'm looking at this and honestly panicking a little because we have people trying to check out right now and getting nothing back. can someone walk me through what the first thing to check should be, like logs or something? sorry if this is dumb im new here

Let me attempt to systematically delineate all non-trivial vectors that could precipitate a 504 Gateway Timeout on what is ostensibly a production API endpoint, while maintaining full awareness of the fact that each vector itself contains several sub-vectors that require their own independent decomposition before we can arrive at a synthesis. First, and perhaps most obviously, we must consider the upstream service dependency chain in its entirety rather than in isolation; if your gateway sits ahead of a microservice mesh where Service A calls B which calls C, a 504 reported to the client is merely the tip of an iceberg — it could originate from any of those hops depending on exactly where the timeout threshold was breached. The upstream proxy (Nginx, Envoy, ALB) has its own keepalive timeouts, while each individual service in your mesh will have its own request and response timeouts configured independently, and a mismatch between these values is one of the most pernicious sources of spurious 504s because the edge gateway may time out before the downstream service's internal timeout fires, masking the true root cause. We should be looking at the specific nginx upstream_read_timeout vs http_proxy_connect_timeout configuration on your ingress controller and comparing it to whatever Go context deadlines or Python Gunicorn timeouts you have set up in each individual pod — if a downstream call returns 504 while its own internal timeout is still running, that's an edge case of upstream proxy misconfiguration.

Secondly, we need to address the database connection pool contention vector which is frequently conflated with general network latency. If your API endpoint executes a query that pins all available connections in the HikariCP or SQLAlchemy pool and new requests are queued beyond the gateway's timeout window, you get a 504 even though no single request actually exceeded its own internal execution budget — it was merely starved of resources before it could begin. This requires an analysis of connection acquisition wait times rather than just overall query duration. We should