NGINX proxy retries: test GET recovery and POST failures

NGINX can retry a failed GET against another upstream when proxy_next_upstream allows it. A POST already sent upstream is treated differently: without non_idempotent, the tested policy returns its 503 response instead of sending that POST to the backup. This matters because the first server may already have applied the write.

The NGINX 1.28.0 lab makes that outcome observable. Backend A returns 503; backend B returns 200. The GET succeeds through B after A fails. The POST increments A’s counter before returning its error, leaving the client with a failed response despite a state change. You will compare response bodies and upstream logs to see exactly which requests were retried.

The NGINX example and test runner are available with the recorded results.

Run the NGINX upstream retry lab

Before you start:

Install Python 3.11 or newer, Git, and Docker. Confirm the Docker engine is running with docker info.
Run the commands from the repository root after cloning. The examples use pinned container versions and download them on the first run.
Use a local development machine with room for disposable containers. The harness removes the resources it creates.

Its harness creates a temporary network, starts both backends, waits for them to answer, and starts NGINX with a loopback-only published port. It removes the containers and network after the assertions.

git clone https://github.com/emminextechdocs/developer-examples.git
cd developer-examples
python3 deep-dives/walkthrough.py nginx

The run saves deep-dives/evidence/nginx.json, including the observed backend and access-log output. The backend counter is in memory and exists only to show that work can happen before an error response. It is not a durable order service or an implementation of business idempotency.

walkthrough.py runs the lab and prints selected commands and their responses alongside the assertions. Open the full-resolution capture.

Inspect the result before changing the example

The runner writes a fresh JSON record and exits after cleanup. To print only its assertions, run:

python3 - <<'PYTHON'
import json
from pathlib import Path
report = json.loads(Path("deep-dives/evidence/nginx.json").read_text())
assert report["passed"] is True
for event in report["events"]:
    if "assertion" in event:
        print(event["assertion"], "=>", event["actual"])
PYTHON

This prints each assertion and its observed value from your latest run. If a check fails, fix its setup or implementation and rerun the lab. The following sections explain what the checks establish.

Create backends with observable failure behavior

Backend A always returns 503 for the test endpoint. For POST, it first increments its counter. Backend B returns 200. Both expose /stats for readiness so the experiment does not confuse a server that has not started with the intended application-level failure.

from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import os

name = os.environ["BACKEND_NAME"]
posts = 0

class Handler(BaseHTTPRequestHandler):
    def respond(self, status):
        body = json.dumps({"backend": name, "posts": posts}).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        self.respond(200 if self.path == "/stats" or name == "b" else 503)

    def do_POST(self):
        global posts
        self.rfile.read(int(self.headers.get("Content-Length", "0")))
        posts += 1
        self.respond(503 if name == "a" else 200)

HTTPServer(("0.0.0.0", 8080), Handler).serve_forever()

The resulting write is ambiguous to the caller: the client sees an error, but the server has already changed state. The response includes the counter so the test can verify the state change despite the error.

The Python server is a controlled fixture. The Python HTTP server documentation describes the library and its production limitations. Replace it with your application’s real transaction boundary before drawing conclusions about durable effects.

Configure a primary and backup upstream

The configuration makes A the primary and B a backup. It sets A’s max_fails to zero so failure accounting does not temporarily remove A after the GET test. That way the subsequent POST still reaches A and tests the intended non-idempotent boundary.

The proxy configuration is deep-dives/nginx/nginx.conf:

events {}
http {
  log_format attempts '$request_method $status upstream=$upstream_addr status=$upstream_status';
  access_log /dev/stdout attempts;
  upstream application {
    server a:8080 max_fails=0;
    server b:8080 backup;
  }
  server {
    listen 8080;
    location / {
      proxy_pass http://application;
      proxy_next_upstream error timeout http_503;
      proxy_next_upstream_tries 2;
      proxy_next_upstream_timeout 2s;
      proxy_connect_timeout 500ms;
      proxy_read_timeout 1s;
      proxy_send_timeout 1s;
    }
  }
}

Check these settings before running the requests:

Backend A stays eligible with max_fails=0, so the later POST still reaches the deliberately failing primary.
Backend B is the backup used when an eligible retry occurs.
http_503 permits retrying a qualifying request after the upstream’s 503 response.
Omitting non_idempotent preserves the boundary tested by the sent POST.

The harness starts both backends, sends the GET and POST, and checks their reported effect counters. It also records the proxy access log.

This is a testing choice, not a proposed production health policy. NGINX’s upstream module documentation describes backup servers and failure-accounting controls. In a real deployment, choose those settings using availability requirements and measured backend behavior.

Without this explicit choice, a failed GET could change which server receives the next request. A POST that happens to reach B and returns 200 would then tell us nothing about whether NGINX retried it after A performed work. Keeping A eligible makes the two request paths comparable.

A read can move from a failing primary to a backup while a sent POST stops at its first response

Verify GET recovery after an upstream 503

The GET reaches A, receives 503, and is tried against B. The final response identifies backend B with a zero POST count. The access log records two upstream addresses and statuses 503, 200.

The result demonstrates the selected retry condition and backup path. It does not prove that all failed GETs are safe to repeat in every application. HTTP defines GET as a safe method, but an application that performs unintended mutations on GET can violate that expectation. The HTTP semantics specification defines method safety and idempotence.

Use this test shape with your actual read endpoint. Confirm both the final response and the number of upstream attempts. A client receiving 200 after recovery is useful, but it can hide repeated work or a slow first attempt unless the attempt sequence is visible.

Verify that a sent POST is not retried

Compare the actual access-log lines from the recorded run. Container IP addresses vary:

GET 200 upstream=172.23.0.2:8080, 172.23.0.3:8080 status=503, 200
POST 503 upstream=172.23.0.2:8080 status=503

The GET has two upstream addresses and two statuses: A failed, then B returned 200.
The POST has one address and one 503. Check the recorded backend response too: posts is 1 on A, so the failure response does not mean the write was absent.

To inspect your run, use the assertion-printing command above and find upstream attempts captured in access log. Do not compare literal container addresses across runs.

The POST reaches A, increments its counter, and returns 503 with posts: 1. The configured NGINX policy returns that failure instead of passing the sent POST to B. The access log shows one upstream status: 503.

NGINX’s proxy module reference documents the separate non_idempotent option and the restrictions on retrying requests already sent upstream. The fixture deliberately does not enable that option.

The result applies to a POST already sent upstream. Connection timing and configuration matter. Our observed case is a POST already received and acted on by A, followed by an application 503. Keep that request timing in mind; a failure before sending any request is a different scenario.

Recover from an uncertain write result

A 503 response from this fixture does not mean the counter stayed unchanged. In a real application, the corresponding uncertainty might concern an order insert, payment authorization, or background-job submission. Returning an HTTP error does not roll back the earlier state change.

A client retry therefore needs a business-level contract. An idempotency key can help if the destination durably associates that key with the operation and defines what repeated requests return. Merely forwarding a header through NGINX does not implement that contract.

Our webhook recovery walkthrough examines the related distinction between request acceptance, acknowledgement, and completed processing. Use that model when documenting whether an upstream error is safely retryable, uncertain, or requires a status lookup before another attempt.

Set attempt limits and understand proxy timeouts

The configuration limits the attempt count and retry window, and it sets connect, read, and send timeouts. These controls address different parts of communication. They should not be summarized as one guaranteed maximum request duration.

In particular, NGINX documents proxy_read_timeout as a limit between successive upstream read operations, not a whole-response deadline. A response that continues producing data is different from one that becomes silent. The fixture’s immediate responses do not test streaming timing.

Build a separate slow-response experiment before adopting the values for production. Measure connection establishment, time to first byte, gaps between response chunks, total client duration, and retries. Those measurements help identify which control actually caused a request to stop instead of attributing every timeout to the same setting.

Handle failures after response streaming begins

Once response data has been sent to the client, retrying another upstream cannot transparently replace what the client already received. A failure halfway through a streamed body is therefore a different recovery problem from a 503 received before the downstream response begins.

This matters for downloads, server-sent events, and other long-lived responses. The application may need resumable operations, reconnect semantics, or an explicit partial-result contract. Increasing the proxy’s retry count does not create those capabilities.

Here, A returns a small complete error before NGINX starts sending the final response to the client. If your endpoint streams, add an experiment that sends some bytes and then fails. Verify what the client actually observes, including incomplete bodies and connection termination.

Check whether a request body can be resent

Whether NGINX can resend a body is another question distinct from whether resending is safe. Buffering affects whether the body can be sent again. The HTTP method and destination behavior determine whether another attempt is safe.

Do not enable a non-idempotent retry simply because a buffered body is available. The first server may already have accepted or committed the operation. Conversely, disabling buffering can change failure behavior for large uploads even when the method is intended to be idempotent.

Review body-size limits and buffering behavior with representative payloads. The HTTP core module documents request-handling settings that interact with the proxy layer. Measure disk use and backpressure separately with representative request bodies.

Log upstream attempts and final status

The custom log format records method, final status, upstream addresses, and upstream statuses. For the GET, the final 200 and upstream 503, 200 tell a different story from a first-attempt 200. For the POST, the single 503 identifies the boundary where the proxy stopped.

The log module reference describes log configuration. Add timing fields when investigating latency, and correlate the proxy request with application logs using an appropriate request identifier.

Do not place credentials, authorization headers, or entire sensitive bodies into the log merely to make retry debugging easier. The fixture needs only synthetic payloads and a counter. A production investigation should use the smallest context that can distinguish attempts and identify the business operation.

Coordinate retries across clients and proxies

NGINX may not be the only component attempting recovery. A client library, gateway, service client, and queue worker can each have its own retry policy. When those layers overlap, one user operation can generate more backend work than any single configuration suggests.

Map the actual request path and assign responsibility for recovery at each boundary. For writes, identify where operation identity is enforced and which component can determine whether the effect already happened. For reads, budget extra attempts against latency and backend capacity.

Document which layer owns each retry decision. Use recorded attempt counts to validate the model. Our API documentation guide discusses why error and recovery semantics belong in the integration contract rather than being left to client guesswork.

Validate configuration before reload

Run the configuration check before applying a change, then verify the behavior after reload. A syntactically valid configuration can still route to the wrong upstream or apply a retry condition more broadly than intended. Keep both the GET and POST tests in the release check.

NGINX’s control documentation describes reload behavior. Do not replace that with an assumption that restarting the process and reloading configuration are interchangeable operational steps. Decide which behavior your deployment system intends to use.

Retain the previous configuration and its test result. If a change increases upstream attempts or changes write outcomes unexpectedly, you need a known comparison point. The beginner’s configuration guide is also useful for understanding how configuration contexts compose, even when the retry problem itself is advanced.

Test durable effects alongside retry counts

An upstream-attempt count is necessary evidence, but it is not the business invariant. For a real write endpoint, add a durable operation identifier and verify how many business records or effects exist after an uncertain response. The first server’s response and the second server’s response may both look reasonable while the combined operation is wrong.

Use a fault point after the durable mutation and before the response is delivered. Then exercise the actual client, proxy, and server retry configuration. Record whether the client performs another request, whether the proxy creates another upstream attempt, and whether the destination recognizes the repeated operation. This separates three layers that are often compressed into the phrase “the request retried.”

For read endpoints, test backend work as well as final latency. Two attempts can be acceptable during a brief failure but expensive during sustained degradation. A retry policy should have an explicit capacity assumption and a way to observe when that assumption stops holding.

The supplied backends return immediately and use an in-memory counter, so the results describe retry behavior rather than production performance. For your application, name the failure point and assert the resulting business effect count.

Frequently asked questions

Does NGINX retry POST requests by default?

Requests with non-idempotent methods such as POST are normally not passed to the next server once the request has been sent upstream. The non_idempotent option changes that policy. A connection failure before sending is a different case. This lab specifically tests a POST that A has received, acted on, and answered with 503.

How can you tell whether NGINX retried a request?

Log the upstream addresses and statuses alongside the final response status. The lab’s GET shows upstream statuses 503, 200, while its POST shows one 503. Also check the backend response or operation record: multiple attempts describe network activity, but they do not by themselves establish how many business effects were committed.

Is proxy_read_timeout a deadline for the whole response?

No. proxy_read_timeout limits the interval between successive upstream read operations. A response that keeps producing data can therefore last longer than that value. Test streaming endpoints with delayed chunks and record both idle gaps and total duration. Keep connect, send, read, and retry settings distinct when diagnosing which timeout ended a request.

Is an Idempotency-Key header enough to enable write retries?

No. The destination must durably associate that key with the operation and define how duplicates, concurrent requests, and payload changes are handled. NGINX forwarding the header only transports the value. Test a failure after the business commit but before the response, then verify that a retry does not create a second effect.

← Back to all articles