The AI Agent Ran Once. Why Did the Server Restart Twice? I Crashed a Worker in the Most Dangerous Millisecond

TL;DR

A user can click once while a queue delivers the task to a Worker twice. The Worker may intend to run once while the outside world receives two restarts, two emails, or even two charges. The dangerous interval is tiny but fundamental: the side effect has succeeded, and the Worker crashes before saving COMPLETED. The queue cannot inspect the outside world, so an at-least-once system redelivers. The answer is not to believe that a broker can magically provide end-to-end exactly-once behavior. It is to combine a durable task state machine, a stable idempotency key, a receiver-side uniqueness constraint, leases, fencing tokens, a transactional outbox, reconciliation, and compensation.

I built a zero-network lab using only Python’s standard library and local SQLite. It first reproduces one intent causing two effects, then proves that two deliveries converge on one safe effect. It also demonstrates lease takeover, rejection of a stale Worker, outbox recovery across two crash points, and compensation when a receiver cannot deduplicate. Everything is synthetic: no model, server, credential, or third-party service is touched. The final acceptance gate passes 7/7 checks. One-click entry points for Windows 11, Ubuntu 26.04, and macOS 26 are included.

Figure 1: Original cover created for this article. “Server restart” is represented by a synthetic row in a local database. The lab never connects to or operates a real machine.

1. Background: One User Click Does Not Mean One Delivery

In the production AI Platform reference architecture, I included a Worker crash in the release gate. That article showed the outcome, but it did not fully open the most dangerous millisecond. This article fills that gap.

Imagine a restaurant. You hand the waiter one ticket that says “one plate of fried rice.” The cook prepares it and places it at the pickup counter. Just before stamping the ticket DONE, the power goes out. When power returns, the system sees an unstamped ticket and prints it again for another cook. The second cook cannot see that the first plate is already waiting, so another plate is prepared.

The customer says, “I ordered once.” The queue says, “I guarantee the ticket is delivered at least once.” Both statements are true, yet two plates are sitting on the counter.

Replace fried rice with a business operation and the story stops being cute:

• service.restart runs twice; the second restart interrupts recovery from the first.
• A welcome email, alert, or support ticket is sent twice.
• Inventory is decremented twice for one order.
• A payment endpoint receives two requests and the customer sees two charges.
• An Agent edits the same file or configuration twice, using the first result as input to the second mutation.

Teams often call this a “duplicate model tool call.” The model, however, may have emitted one tool intent. What repeated was the delivery and side-effect protocol of a durable task. Looking only at prompts and conversation history therefore fixes the wrong layer.

Figure 2: The Worker has crossed the outside world’s point of no return while the local task still says RUNNING. The process may disappear for one millisecond, but the uncertainty remains until the system reconciles it.

2. Symptoms: Every Log Line Can Look Like a Healthy Retry

A typical incident looks deceptively reasonable. There is one task_id. A first Worker claims it, the external endpoint accepts the operation, and then the Worker loses its process, database connection, or power before persisting completion. Its lease expires. A second Worker claims the same task, calls the same tool, and successfully stores COMPLETED.

If the logs contain only “start, failure, retry, complete,” Operations sees a beautiful recovery sequence: attempt one failed and attempt two succeeded. The user sees two actions. A successful task state does not prove that the business effect happened once.

In the naive path of my lab, the injected fault happens after action=ACCEPTED and before COMPLETED. Delivery one creates one synthetic restart. Recovery redelivers the task, and delivery two creates another:

Figure 3: Real local lab-output capture. Delivery one is accepted and then crashes. The queue redelivers, delivery two is accepted, and observed_side_effects=2. All behavior is local and synthetic.

The window is not limited to a dramatic process crash. It can also appear when:

• a tool returns success but the task database is temporarily unwritable;
• the completion transaction loses its connection, so the Worker cannot know whether it committed;
• the queue acknowledgment is lost and the broker considers the message outstanding;
• execution exceeds its lease and another Worker takes over;
• a Worker experiences a long garbage-collection pause, sleep, or network partition, then wakes believing it still owns the task;
• an Agent receives cancellation after the external request has passed its cancellable point.

All of these share one property: a local Boolean cannot simultaneously prove “I remembered completion” and “the outside world did this only once.”

3. Root Cause: Delivery, Attempt, and Business Outcome Were Treated as One Thing

3.1 At-Least-Once Delivers the Ticket; It Does Not Promise One Plate

Reliable queues normally prefer a duplicate over a lost task. If a message is not acknowledged in time, a consumer disconnects, or a lease expires, the broker delivers it again. That is at-least-once delivery. It means the message will have another chance to be processed. It never promised that a business action would happen only once.

At-most-once chooses the opposite tradeoff: do not deliberately redeliver, even if an operation may disappear. That can be acceptable for a disposable cache refresh. It is normally unacceptable for payments, approvals, deployments, and infrastructure actions. Production systems therefore often keep at-least-once delivery and make the business effect idempotent.

3.2 “Exactly Once” Is Incomplete Until You Name the Boundary

Some databases and streaming systems provide exactly-once semantics inside their own log, transaction, or consume-process-produce boundary. An Agent task typically crosses a queue, task database, HTTP tool, email provider, payment system, and perhaps a physical device. Those systems do not share one atomic transaction.

So “we enabled exactly once on the queue” is not an answer. Ask four more questions:

1. Does it deduplicate broker records, or does it prevent the external API from causing a second effect?
2. What happens when a Worker crashes after the external success but before its acknowledgment?
3. When an external request times out, was it rejected, or did it succeed and lose the response?
4. How long are deduplication records retained, and will a very late retry still recognize the key?

A practical end-to-end objective is: a task may arrive more than once, but all attempts for one business intent converge on at most one accepted effect. If the outcome is ambiguous, stop blind retries and move to query, reconciliation, or compensation.

3.3 Conversation History Is Not Business State

A model message saying “the restart completed” is text, not a fact record. A reliable task must distinguish at least PENDING, LEASED/RUNNING, COMPLETED, UNKNOWN, and RECONCILING. The task remains recoverable after a chat closes, context is compacted, or the model is replaced.

Figure 4: Simplified state machine. A production implementation may add WAITING_APPROVAL, FAILED_RETRYABLE, FAILED_TERMINAL, CANCEL_REQUESTED, CANCELLED, and COMPENSATING. Every transition still needs a precondition and version.

This is also a central lesson in 12-Factor Agents: execution state and business state must line up so an Agent can pause, resume, and be audited. Hiding state inside a prompt merely disguises a database problem as a language problem.

4. First Defense: Generate a Stable Idempotency Key for the Business Intent

Idempotency does not mean “never retry.” It means repeating the same request produces the same business result as running it once. The simplest analogy is a parcel tracking number. A parcel may be scanned twice at a sorting center, but the warehouse ships one box.

A useful idempotency key normally binds:

• the tenant or business scope;
• an intent_id or task_id generated once at the entry point;
• the tool name or action type;
• canonical arguments with stable ordering, whitespace, and units;
• the tool schema or business-semantics version.

One option is to hash canonical JSON:

canonical = json.dumps({
    "tenant": tenant_id,
    "task": task_id,
    "action": tool_name,
    "arguments": normalized_arguments,
    "version": "v1",
}, sort_keys=True, separators=(",", ":"))
key = hashlib.sha256(canonical.encode("utf-8")).hexdigest()

Do not include attempt, current time, a random nonce, or a changing trace ID. If the key changes on every retry, every retry looks like a new parcel. Do not use only the tool name either, or two legitimate restarts on different days could suppress each other.

4.1 The Uniqueness Constraint Belongs Near the Effect Receiver

An already_done = true variable inside the Worker is useless after the process dies. A local “processed” row inserted before the external call is also dangerous. If that row commits and the Worker crashes before calling the external endpoint, recovery may treat the marker as success and permanently skip the operation.

The strongest design lets the component that causes the effect accept an idempotency key and atomically return the result of the first request. That may be a payment provider’s request key, a business unique index on an order table, a conditional object-store write, a resource-version compare-and-swap, or an Inbox table in a tool adapter.

The local lab treats a SQLite table as the synthetic effect receiver and makes idempotency_key its primary key. The first request inserts the result and then crashes. The second request carries the same key, so it retrieves the first result instead of executing again:

Figure 5: Real local lab-output capture. Both deliveries have the same key. The first is EXECUTED, the second is DEDUPLICATED, and only one effect exists when the task reaches COMPLETED.

4.2 An Idempotency Store Is Not a Permanent Junk Drawer

The record should include the key, request digest, state, result digest, first-seen time, completion time, and expiry policy. Reusing one key with different arguments must return a conflict, not silently return a result for the wrong request. Retention must cover the queue’s maximum retry and delay, the manual recovery window, and any disaster-replay horizon. Payments, inventory, and regulated actions may need a longer business-reconciliation period.

An IN_PROGRESS record also needs semantics. A recovery Worker cannot wait forever, and it cannot simply repeat the operation. It should combine the lease, receiver query, and recorded outcome to choose between waiting, taking over, querying, or entering UNKNOWN.

5. Second Defense: Leases Enable Takeover; Fencing Tokens Reject the Resurrected Worker

A lease resembles a library card. Worker A owns the task for a bounded interval and renews it with heartbeats. Once it expires, Worker B can take over. The expiration does not make Worker A’s process vanish. It may be paused, partitioned, asleep, or waking up, then continue to execute.

With only a lease, A and B can briefly believe they may write. A lease tells Workers who should work; a fencing token tells the receiver whose work is still valid. Every takeover receives a monotonically increasing generation such as 41 and 42. The receiver remembers the largest generation it has accepted and rejects anything smaller.

Figure 6: The real guard stands at the receiver. Telling the old Worker “your lease expired” is insufficient because it may not hear the message. The resource must enforce a conditional write.

The lab uses a logical clock rather than wall time. Worker A gets token 1 at t=100, with a lease ending at 105. At t=106, Worker B takes over and receives token 2. B’s write is accepted. A then wakes with token 1, and the receiver returns REJECTED_STALE:

Figure 7: Real local lab-output capture. worker-a and worker-b are synthetic role labels, not machine names. Only the current owner can conditionally complete the task.

Three production details matter. Lease acquisition and token increment must be atomic. Time comparisons should happen in one authoritative store rather than on drifting machine clocks. The receiver must actually compare the token. Printing a generation number in a log does not create a fence.

6. Third Defense: Transactional Outbox Closes “Business Committed, Message Missing”

Another gap appears between a task database and a messaging system. Business state commits, and the process crashes just before publishing the event. Reversing the order creates the mirror-image failure: the event is visible while the business transaction rolls back.

The Transactional Outbox pattern works like a restaurant cash register that records both the paid order and a “tell the kitchen” ticket in the same ledger transaction. A separate Dispatcher scans unsent tickets, publishes them, and marks them sent. A committed business change can no longer permanently lose its event.

Figure 8: Outbox prevents loss. Receiver deduplication by event_id prevents duplicate effect. Production systems normally need both.

The lab injects two failures. First it crashes after the local transaction commits but before publication. A Dispatcher recovers and sends the event downstream, then crashes after downstream acceptance but before setting sent. Dispatch two necessarily sends again, but the downstream unique event_id deduplicates it:

Figure 9: Real local lab-output capture. There are two dispatch attempts and one downstream effect. The Outbox Dispatcher is itself at-least-once, so a downstream Inbox or idempotency key is still required.

Outbox is not a universal distributed transaction. It cannot automatically make an endpoint safe if that endpoint offers no query, idempotency, or reversal. It keeps local facts and pending events together; the end-to-end receiving protocol remains a design responsibility.

7. What If the External System Cannot Deduplicate? Query First, Reconcile Next, Compensate Last

Some legacy systems accept no idempotency key and expose no conditional version. After a timeout, they may not even offer a reliable result query. The most dangerous response is “timeout means failure, so retry the same mutation.” A timeout means you did not receive an answer. It does not mean the other side did nothing.

A safer order is:

1. Query by business order, resource version, or bounded time window.
2. If a person can resolve it, move the task to UNKNOWN / RECONCILING and pause automatic retries.
3. If a duplicate happened and the operation is reversible, issue a separately audited compensation.
4. For irreversible actions, use manual intervention, impact limits, and stronger approval before execution.
5. Push the receiver toward an idempotency key or conditional write instead of making the caller guess forever.

The lab models a reservation receiver with no idempotency support. Attempt one creates a reservation but loses the acknowledgment. A blind retry creates another. Reconciliation finds two ACTIVE records, and compensation cancels the duplicate so one remains:

Figure 10: Real local lab-output capture. Compensation limits damage; it is not exactly once. An email cannot be “unread,” and a refund does not erase the customer’s experience, so receiver-side idempotency remains preferable.

8. A Production Execution Protocol You Can Implement

Put the pieces together and a high-risk tool call can follow this protocol:

1. Verify user, tenant, and permission at the entry point; generate a stable intent_id for one business intention.
2. Let the model propose a structured tool call, never execute it directly. See Tool Calling and MCP for that boundary.
3. Have deterministic policy normalize arguments, classify risk, and create an argument- and version-bound approval when necessary.
4. Create the task and Outbox event in one transaction; place the task in PENDING.
5. Let a Worker atomically acquire a lease and increasing fencing token; transition to RUNNING.
6. Generate a stable idempotency key in platform code. Do not ask the model to invent one.
7. Send key, fence, deadline, and narrowly scoped credential through the tool adapter to the receiver.
8. On a definite success, save a result digest and conditionally transition the current generation to COMPLETED.
9. On a definite retryable failure, respect backoff and retry budget while reusing the same key.
10. On an ambiguous outcome, enter UNKNOWN, then query, reconcile, or compensate. Never retry blindly.
11. Record trace, audit, and metrics across the chain without logging secrets or raw sensitive results.

Keep three identifiers distinct. intent_id means one thing the user wants done. attempt_id means one execution attempt. idempotency_key says which attempts must converge on one business result. Treating an attempt as a new intent turns every retry into a new operation.

9. Observability: Delivery May Increase; Duplicate Business Effect Must Stay at Zero

Traditional monitoring often stops at queue consumption success, task completion, and retry counts. None of those proves the outside world was modified once. Observe at least:

• task_delivery_total: delivery count, which may exceed task count during recovery;
• tool_attempt_total: actual attempts sent toward a tool;
• idempotency_hit_total: repeated requests that returned an existing result;
• accepted_business_effect_total: effects confirmed by the receiver;
• unknown_outcome_total: results that are neither definite success nor definite failure;
• lease_takeover_total and stale_fence_rejected_total;
• outbox_oldest_pending_seconds and dispatch retries;
• compensation_total, compensation success, and reconciliation wait time.

Evidence should connect task, intent, attempt, an idempotency-key digest, tool version, fence, logical lease owner, outcome source, and trace ID. Do not trade privacy for apparently rich logging by adding full arguments or actual machine identities.

Figure 11: Real local metrics.json capture. The naive path creates two effects while the safe path creates one. Outbox produces one downstream effect; network and real-credential counters are both zero.

The most useful SLO is not “no retries.” It is: redelivery may occur; duplicate accepted business effect remains zero; ambiguous outcomes are reconciled within a defined time; late writes from old Workers are rejected.

10. One-Click Lab on Windows 11, Ubuntu 26.04, and macOS 26

The complete worker_crash_lab.py uses only sqlite3, hashlib, json, and filesystem APIs from Python’s standard library. It never accesses the network, reads environment variables, loads a credential, or calls a real tool. See the README for the exact boundary.

10.1 Windows 11 PowerShell

Download Run-Windows11.ps1 and the Python file into the same directory, then run:

Set-ExecutionPolicy -Scope Process Bypass
.\Run-Windows11.ps1

The script prefers the Windows Python Launcher through py -3 and falls back to python.exe. It does not install software or contact a third-party service.

10.2 Ubuntu 26.04

Download run-ubuntu-26.04.sh and the Python file into one directory:

chmod +x run-ubuntu-26.04.sh
./run-ubuntu-26.04.sh

10.3 macOS 26

Download run-macos-26.sh and the Python file into one directory:

chmod +x run-macos-26.sh
./run-macos-26.sh

All three entry points run the same standard-library lab and require seven [PASS] lines plus result=PASS. The --clean option may remove only a result directory named lab-output or reference-run, deliberately bounding cleanup.

10.4 Manual Automated Execution

Without a platform entry point, run:

python3 worker_crash_lab.py --clean

Open lab-output/results/06-acceptance-summary.txt and require 7/7. Then inspect the reference-run metrics.json: naive_side_effects=2, safe_side_effects=1, downstream_effects=1, and network_calls=0. [EXPECTED RISK] in the naive scenario is not a failed lab; it proves that the dangerous control case was genuinely reproduced.

10.5 Agent-Automated Configuration and Acceptance

Give AGENT-PROMPT.md to an Agent allowed to execute local commands. It requires the Agent to inspect the files and safety boundary before selecting the current platform entry point. A failure must preserve evidence and explain the cause; editing output to manufacture a PASS is forbidden.

A shorter instruction is:

Read the README, Python lab, and platform entry point. Confirm zero network, zero credentials, and writes limited to lab-output. Run the Windows 11, Ubuntu 26.04, or macOS 26 entry point as appropriate. Require 7/7 acceptance, naive effects equal to 2, safe and downstream effects equal to 1. Stop and report the real cause on failure; never modify acceptance output.

Agent automation is like asking a highly motivated apprentice to run an experiment: provide the bench, boundaries, and acceptance sheet instead of merely saying “prove idempotency works.”

11. Acceptance Result: What the Lab Proves—and What It Does Not

Figure 12: A release gate checks a business invariant, not merely HTTP 200, process liveness, or an empty queue.

The reference run used a local Python 3 standard library and passed all seven checks: the danger window is reproducible; two safe deliveries produce one effect; a stale Worker is fenced; two Outbox dispatches produce one downstream effect; compensation leaves one valid reservation; safe tasks reach a terminal state; and privacy/network counters remain zero.

Figure 13: Real local lab-output capture, 7/7 PASS. The Python version shown belongs only to this reference run. The lab does not pretend to replace a real broker, external API, concurrent load, or production failure drill.

It proves protocol logic and an acceptance method. It does not prove that SQLite is a production queue, nor that a local transaction can wrap every remote tool. Before launch, add multi-process concurrency, real broker redelivery, database failover, long network partitions, receiver throttling, idempotency-record expiry, approval timeout, and disaster-recovery tests.

12. Q&A

Q1: The Queue Promises Exactly Once. Do I Still Need an Idempotency Key?

Yes. First identify the covered boundary. Broker-level consume/produce semantics do not guarantee that email, payment, device, or arbitrary HTTP endpoints create one effect. If the outside world is not in the same transaction, receiver idempotency or a business uniqueness constraint is still required.

Q2: Why Not Mark the Task COMPLETED Before Executing?

That changes duplication into loss. If the Worker stores completion first and crashes before the tool call, the system permanently believes the action happened. Record an honest in-progress state, then rely on receiver idempotency, query, and recovery semantics.

Q3: Can the Model Generate the Idempotency Key?

It should not. A model may change format, omit a field, or generate a fresh value on retry. Deterministic platform code should derive it from verified identity, business intent, and canonical arguments. The model may propose business arguments, not the protocol identity.

Q4: Is a Unique Constraint on the Task Table Enough?

No. It prevents duplicate task creation but cannot prove what happened at a remote receiver. Put uniqueness as close as possible to the true side effect, or use the receiver’s native request key and outcome query.

Q5: Why Not Kill the Old Worker When Its Lease Expires?

During a partition or process pause, the control plane may not be able to kill it promptly. Even after a termination signal, an in-flight request may arrive. A fencing token lets the receiver reject the old generation without depending on the old Worker cooperating.

Q6: Does Every Tool Need the Same Idempotency Protocol?

No; risk differs. Pure reads may use caching and duplicate suppression without an external mutation. Overwrite, append, and irreversible writes need increasingly strict treatment. Duplicate notifications and duplicate transfers are both duplicates, but their impact and compensation options are radically different.

Q7: Does Compensation Mean the Operation Never Happened?

No. A refund may incur a fee, an email cannot be unread, and an extra restart’s outage cannot be erased by another restart. Compensation is a separate, audited business correction. It should never be marketed as exactly once.

Q8: How Long Should Idempotency Records Live?

At least through the maximum queue delay, retry backoff, manual recovery, and disaster replay window, then according to reconciliation and regulatory needs. Deleting too early makes a late retry execute again; storing forever creates capacity and privacy costs. Retention should be risk-tiered by action.

Q9: Can Cancellation Stop an In-Progress Side Effect?

Only before the external operation crosses its commit point and only when the protocol supports cancellation. Distinguish CANCEL_REQUESTED from CANCELLED. If the outcome is already ambiguous, reconcile it rather than promising the user that it was canceled.

Q10: Which Faults Are the Minimum Pre-Launch Set?

Inject a crash after effect, lost acknowledgment, old Worker revival after lease takeover, ambiguous database commit, Outbox delivery before the sent flag, downstream timeout with actual success, idempotency-key conflict, and idempotency expiry. For every case, inspect both task state and receiver-side effect count.

13. Continue Reading and References

To place this mechanism inside the whole platform, continue with the AI architecture practice reading map. The full system is in the production AI Platform reference architecture; the tool boundary is covered in Tool Calling and MCP; state and control-flow principles are in 12-Factor Agents.

Further reading:

• AWS Builders’ Library: Making retries safe with idempotent APIs
• Stripe API: Idempotent requests
• Microservices.io: Transactional Outbox pattern
• Apache Kafka documentation: Message delivery semantics
• Martin Kleppmann: How to do distributed locking

The release gate fits in one sentence: the system may inspect the same ticket more than once, but the kitchen, warehouse, bank, and production environment must recognize one business intent. If the platform cannot yet prove the outcome, it must stop and reconcile rather than gamble with another retry.