API & Backend
Redis Streams recovery with XAUTOCLAIM and XACK
To recover an unacknowledged Redis Streams entry, use XAUTOCLAIM to transfer an eligible pending delivery to another consumer, then call XACK after the required work is durable. A consumer reading new entries with XREADGROUP ... > does not automatically receive another consumer’s pending messages.
That distinction matters when a worker receives an order and disappears before acknowledging it. This Redis 8.2.1 lab leaves an event with worker A, performs a clean Redis restart using append-only persistence, and has worker B reclaim the same event. You will inspect the pending entries list, follow the returned ID, and confirm that acknowledgement clears pending state while retaining the stream entry. The restart and reclaim checks cover different parts of recovery.
The Redis example and test runner are available with the recorded results.
Run the Redis Streams recovery lab
Before you start:
- Install Python 3.11 or newer, Git, and Docker. Confirm the Docker engine is running with
docker info. - Run the commands from the repository root after cloning. The examples use pinned container versions and download them on the first run.
- Use a local development machine with room for disposable containers. The harness removes the resources it creates.
The harness starts its own Redis container, publishes no Redis port, and removes the container afterward. It uses append-only persistence with appendfsync always for this small experiment. This setting is chosen for the small persistence test; measure its write cost before adopting it.
git clone https://github.com/emminextechdocs/developer-examples.git
cd developer-examples
python3 deep-dives/walkthrough.py redisThe run writes deep-dives/evidence/redis.json. The generated stream ID will differ on your machine. The assertions compare IDs across steps to verify that recovery uses the original event. Preserve this check when adapting the example; a replacement entry with the same payload would not demonstrate recovery of the pending delivery.
walkthrough.py runs the lab and prints selected commands and their responses alongside the assertions. Open the full-resolution capture.
Stream entries and consumer-group progress
A stream stores entries. A consumer group tracks how its consumers receive and acknowledge those entries. Once an entry has been delivered through a group without acknowledgement, it belongs to the group’s pending entries list, commonly called the PEL. That state is separate from the entry remaining in the stream.
The distinction creates two independent questions: is the event payload still retained, and does this group still consider the delivery unfinished? Acknowledgement answers the second question. Retention policies answer the first. Redis documents the pending-entry behavior in XREADGROUP.
Treat the group name as part of the processing contract. Two groups can consume the same stream for different purposes. Clearing pending work for the billing group does not establish that the analytics group has completed its own processing. Design retention with every required consumer in mind.
Step 1: Create the stream and consumer group
Start a fresh Bash session for the manual sequence. The automated run above has already cleaned up its container.
bash
source deep-dives/redis/session.shThe setup starts Redis 8.2.1 with append-only persistence, waits for PING, and defines redis_lab_cli. It publishes no database port and removes its container when this Bash session exits. Keep this session open so the container name and event ID remain available.
The fixture creates an empty orders stream and a workers group starting at ID zero. MKSTREAM lets the command create an empty stream when necessary. Starting at zero allows the group to read existing entries; starting at $ instead starts after the stream’s current last entry. The group creation reference defines those choices.
redis_lab_cli XGROUP CREATE orders workers 0 MKSTREAM
event_id=$(redis_lab_cli XADD orders '*' order_id ord-42)
redis_lab_cli XREADGROUP GROUP workers worker-a COUNT 1 STREAMS orders '>'XGROUP returns OK. The read prints the stream name, generated ID, and order_id / ord-42 pair. The quotes prevent Bash from expanding * or treating > as redirection. Save the generated ID in event_id; every recovery and acknowledgement step must refer to this same entry.
The helper sends these commands through redis-cli --raw. The automated harness uses JSON output for its assertions. A production bootstrap should decide what to do when the group already exists. Deleting and recreating the group merely to make setup succeed would change delivery state and invalidate the recovery assumptions.
Keep consumer identities distinguishable in logs. Reusing one name across unrelated live processes makes ownership harder to interpret operationally. The name is an application-selected identifier, not proof that a unique process is alive or that its work has finished.
Step 2: Compare new reads with pending work
Ask worker B for a new entry:
redis_lab_cli XREADGROUP GROUP workers worker-b COUNT 1 STREAMS orders '>'The raw CLI prints no entry. Compare that with redis_lab_cli XPENDING orders workers: its first line is 1. The two commands inspect different delivery states.
Worker A receives the event with XREADGROUP ... > and intentionally leaves it unacknowledged. Worker B then issues the same new-message read. The response is null because no additional group-undelivered entry is available. The existing pending event has not become new again simply because another consumer asked for work.
This is the first negative recovery check in the lab. It prevents a common recovery mistake: running more consumers while leaving the pending backlog untouched. Adding capacity can improve processing of new entries without addressing entries already assigned to missing workers.
Do not interpret a null result as “the queue is empty.” In this fixture it means the selected read returned no new message. Monitor new entries and pending deliveries together. Your application’s report should name those populations separately rather than hiding them behind a single queue-length number.
Step 3: Check pending state after a Redis restart
Restart the same container, then inspect the group:
docker restart "$redis_lab"
wait_redis
redis_lab_cli XPENDING orders workersIgnore the container name printed by docker restart. The XPENDING response starts with count 1, followed by the minimum and maximum pending IDs and the consumer summary. Both IDs must match event_id, and worker A still owns that pending delivery.
The harness performs a clean container restart, waits for Redis to answer again, and calls XPENDING. The summary reports one pending entry. The pending state survived the clean restart. Disk loss and replica promotion require separate tests.
Redis’s persistence guide explains the difference between snapshotting and append-only logging, including the trade-offs among fsync policies. Our experiment uses the same container storage across the restart. Removing that container and creating an unrelated one would be a different storage test.
For production, write down which failure you are designing for: a consumer disconnect, a Redis process restart, a host failure, or a failover. A passing test for one of those events is useful evidence, but it is not interchangeable with the others. Recovery documentation should identify the storage and replication assumptions next to the commands.
Step 4: Reclaim the original entry with XAUTOCLAIM
Worker B calls XAUTOCLAIM using a minimum idle time of 100 milliseconds. That small interval keeps the experiment quick. The harness polls until the entry meets the threshold, then checks that the claimed entry has the original ID. The returned entry also contains the order payload.
sleep 1
redis_lab_cli XAUTOCLAIM orders workers worker-b 100 0-0 COUNT 10The one-second wait ensures this single entry has passed the deliberately short idle threshold. In the raw output, expect cursor 0-0, the original entry ID, then order_id and ord-42. The automated version checks the structured response and compares the returned ID directly.
The command returns a scan cursor, claimed entries, and, in the tested version, a list of IDs removed from pending state because their stream entries no longer exist. The XAUTOCLAIM reference defines that response. Parse the structure rather than assuming the whole response is a message list.
A real recovery loop must continue from the returned cursor. Receiving fewer entries than the requested count does not by itself prove that every eligible pending entry has been examined. The fixture has only one event, so its scan is intentionally simple. Preserve cursor handling when adapting the command to a larger backlog.
Choose a reclaim threshold for slow workers
An idle pending entry may belong to a crashed worker, but it may also belong to a slow or paused worker. Reclaiming it changes Redis ownership; it does not cancel code already executing in the old process. Both workers might therefore reach the business operation.
Select the idle threshold using observed processing time and the cost of overlapping execution. If ordinary processing regularly exceeds the threshold, recovery will become a source of duplicate attempts. Increasing the threshold reduces that overlap but delays recovery of genuinely abandoned work. Choose that trade-off for each workload.
A destination that enforces a generation or ownership token can reject stale execution. An idempotency key can address repeated application of the same business operation. Neither is automatically created by reclaiming a stream entry. Our PostgreSQL webhook idempotency guide discusses the database boundary where duplicate effects need to be controlled.
Step 5: Acknowledge and check retention
After reclaiming the event, the fixture acknowledges it with XACK. The first acknowledgement returns one. Repeating the acknowledgement returns zero. The group then has no pending entries, while XLEN orders still returns one.
redis_lab_cli XACK orders workers "$event_id"
redis_lab_cli XACK orders workers "$event_id"
redis_lab_cli XPENDING orders workers
redis_lab_cli XLEN ordersRead the outputs in order:
- First
XACK:1, because one pending delivery was removed. - Repeated
XACK:0, because that delivery is no longer pending. XPENDING: count0; there is no unfinished delivery in this group.XLEN:1; acknowledging did not delete the retained payload.
Run exit when finished to remove the manual container. To repeat the full sequence with ID and count assertions, run bash deep-dives/redis/manual.sh from the repository root.
The XACK documentation explains that acknowledgement removes entries from the PEL. Our length assertion makes the separate retention behavior visible. If your application wants old payloads removed, that requires its own retention policy; it is not an implicit effect of acknowledgement.
In a real worker, acknowledge only after the required work has been committed. That might be the committed business mutation or a committed inbox record that an independent worker will process. Acknowledging immediately after an in-memory handoff creates a loss window if the receiving process exits before persisting the work.
Recover after losing an acknowledgement response
Suppose the business database commits, then the worker loses its connection before Redis receives XACK. Recovery will see an unfinished pending delivery even though the business effect already happened. Reprocessing without deduplication can apply that effect again.
The reverse ordering is also problematic: acknowledging first and committing the business operation afterward can lose the operation if the worker fails between them. Redis and an unrelated database do not become one transaction because the commands appear consecutively in application code.
Choose an explicit cross-system strategy and test both windows. For example, a durable business key with a unique constraint can make a repeated database operation detectable. That is an application design suggestion, not a claim that this stream fixture implements a transactional inbox. Keep the redelivery test while adding a separate assertion for the business effect count.
Set retention long enough for recovery
Pending state does not guarantee that the underlying payload will remain available forever. Trimming or deleting entries can leave pending references whose data is no longer present. Newer Redis versions add more retention-related options, so check the exact version and command semantics rather than copying a retention command from an unrelated deployment.
The XTRIM reference describes length- and ID-based trimming. Choose a retention horizon that reflects processing lag, retry duration, and audit requirements. A shorter stream may hide data loss if trimming removed entries still needed for recovery.
Before changing retention, inspect both lag and pending age for every required group. Create a separate destructive fixture that trims an unacknowledged entry, then verify how your client handles the missing payload and returned deleted-entry IDs. Do not run that experiment against a shared production stream.
Inspect pending age and delivery attempts
XPENDING offers both a summary and a detailed form. The detailed output can expose ownership, idle time, and delivery counts, while filtering lets you inspect a bounded subset. The command reference is the basis for implementing those inspection tools.
Use bounded inspection rather than fetching an entire large pending population for every dashboard refresh. Useful metrics include the oldest pending age, number reclaimed per interval, acknowledgement rate, and repeated delivery count. Correlate those with downstream latency before changing worker concurrency or reclaim thresholds.
Avoid high-cardinality event IDs as metric labels. Put them in logs or traces where individual investigation belongs. Our OpenTelemetry walkthrough shows how request context can support that investigation without turning every individual event into a separate metric series.
Why XACK cannot fence out an old worker
XACK receives the stream, group, and entry IDs. It does not require the name of the consumer that currently owns the pending entry. When an old worker resumes after a reclaim, the application cannot assume acknowledgement is automatically rejected because ownership changed.
Reclaiming lets another consumer take over; it does not stop the previous process. If stale completion would violate a business rule, enforce that rule at the durable destination. Check whether the operation already committed before allowing a replacement or resumed worker to apply it again.
Add a regression case that pauses worker A after it starts the business operation, lets worker B reclaim the entry, and then resumes A. Measure both the effect count and acknowledgement outcomes. A test that merely checks which consumer name appears in pending metadata will miss the important application race.
Keep the operation identity independent of the worker identity. A replacement worker should be able to recognize the same order operation using the original business operation ID. Conversely, a new consumer name should not cause the application to generate a new business idempotency key for the same event. Reusing the operation ID lets the destination recognize a retry.
Frequently asked questions
Why does XREADGROUP return no new messages when XPENDING is nonzero?
XREADGROUP ... > requests entries not yet delivered to the group. XPENDING describes entries already delivered but not acknowledged. Both results can be correct at the same time. Inspect the pending entries and run the recovery path for eligible work instead of assuming that adding consumers will make those deliveries appear as new messages.
What does XAUTOCLAIM return?
On the tested Redis 8.2.1 release, XAUTOCLAIM returns a scan cursor, the claimed entries, and IDs removed from pending state because their stream data no longer exists. Continue scanning from the returned cursor. An empty or short batch alone does not establish that the scan has reached every pending entry eligible for recovery.
Does XACK delete the message from the stream?
No. XACK removes the delivery from that group’s pending entries list. The lab verifies this with pending count zero and stream length one after acknowledgement. Choose a separate deletion or trimming policy for retained payloads, allowing enough time for required groups to process and recover their work. Each group has its own progress.
Does AOF prevent data loss during Redis failover?
A passing AOF-backed restart test does not establish that guarantee. The lab restarts one container using the same storage. Redis replication is asynchronous by default, and failover introduces different loss windows. Review the replication documentation with your actual persistence and promotion settings, then test those failures separately from consumer recovery.