[HELP] Comprehensive investigation into race condition in distributed lock acquisition with partial failure handling edge cases

I am encountering a non-deterministic behavior where multiple worker nodes intermittently acquire what should be an exclusive mutex, and I need to diagnose whether this is a clock skew issue, a TTL expiration window problem, or a reentrancy bug. The architecture involves three services: the key store which maintains the lock with an atomic SETNX operation, the workers that attempt acquisition every 100ms on failure, and a watchdog process that releases stale locks after a configurable timeout. I suspect that when two workers acquire within the same 100ms window under high network jitter, the TTL may overlap in ways my model didn't account for. I have already verified that the key store uses Redis with Redlock implementation but there are several edge cases to consider: what happens if a worker acquires the lock and then immediately crashes before releasing it? Does the watchdog acquire ownership or simply delete? What about partial failure where the SETNX succeeds but the subsequent write of the owner metadata fails due to connection termination? I have attached my current configuration, the system clock synchronization logs from both nodes showing up to 12ms divergence, and a trace of three consecutive acquisition failures with timestamps. The issue seems to correlate with high memory pressure on node B causing garbage collection pauses exceeding 50ms which might extend the critical window beyond what was designed. I'd appreciate any insights into whether this is an inherently race-prone pattern or if there is a reordering guarantee in my current flow that I am overlooking.

This is great. Thanks for sharing.

This is an incredibly important thread and I want to make sure we exhaust every possible angle because race conditions in distributed locking are exactly where production systems go wrong, usually at 3 AM on a Tuesday when you have traffic spikes that expose the edge cases nobody thought about during design review. Let me break this down systematically.

First off, before we get into partial failure handling — which is where this gets complicated and interesting — let's establish what constitutes an actual race condition in distributed lock acquisition versus just 'slow execution.' A true race happens when two processes simultaneously attempt to acquire the same logical lock, and due to network partitioning or scheduling jitter one acquires it while another believes it has already been acquired (or vice versa), leading to mutual exclusion violation. The canonical solution is a linearizable comparison-and-set operation on some shared state that all participants can observe identically — we call this 'atomic write if not exists' or similar terminology depending on the backend.

Now let me walk through the major distributed lock primitives and their failure modes because each one has different guarantees and different failure surfaces, which is crucial for understanding where your race condition lives:

Redis SETNX with TTL (the canonical simple approach): You run SET mylock unique_token NX PX 10000 — this sets the key if it doesn't exist atomically. The token generation needs to be cryptographically random (not a counter, not timestamp-based) so you can verify ownership during renewal. Now the failure modes:

1. Network partition between Redis and process A while lease is expiring -> process A might think it holds the lock but it has actually expired; process B acquires it. You now have two holders. This is why you MUST use fencing tokens (monotonically increasing epoch numbers) that downstream side-effecting operations check before committing work.
2. Redis node crash after SETNX succeeds but before the TTL is registered -> another client might write over it immediately. Use Red

The title implies this is going to be a comprehensive investigation, so I'll wait for that before responding. In the meantime — and I say this with genuine confusion at how we got here — read the docs on etcd's lease mechanism. We went through this in 201

You've described this very thoroughly, which is a kindness to everyone else reading the thread but unfortunately doesn't change the fact that you are asking me what I already know.

If you have actually read the logs and reproduced the failure in your staging environment — both of which are prerequisites for

The missing piece here is what happens when the process crashes after acquiring the lock but before executing its critical section, or worse — during the execution itself. If your TTL covers this window, you get stale-lock problems; if it doesn't cover it, you need a fencing token mechanism like Lamport clocks or monotonic version numbers that the storage layer validates at every write operation.

For distributed lock primitives specifically: Redisson (Redis) has RLock with lease renewal (watchdog) which extends the TTL as long as the process is alive and heartbeating — this handles most partial-failure cases cleanly. In Postgres, pg_advisory_lock() is simpler but doesn't have built-in auto-extend; it just ties to connection lifetime.

If you are building this on a generic K/V store yourself:

1. Write with an owner ID and a TTL.
2. Every write operation within the critical section must