An AI Platform Is Not a Chat Box: I Injected Rate Limits, Worker Crashes, and Stale Memory to Find the Production Architecture

TL;DR

A chat demo needs a model, a prompt, and a web page. An AI Platform trusted to touch business systems must also answer questions about identity, authorization, approval, task state, idempotency, memory, tracing, cost, and recovery. The model proposes a next step; the platform decides whether it may happen, who performs it, how it stops when something fails, whether a retry repeats the action, and how the decision can be explained later. To test those boundaries, I built a dependency-free Python fault lab with zero network calls and zero real credentials. It makes the primary model route return a synthetic 429, holds a high-risk restart at an approval gate, rejects a prompt injection before model or tool execution, crashes a Worker after its side effect but before its checkpoint, and proves that recovery does not execute the effect twice. It also expires one memory and verifies that the stale value never enters context. The six acceptance checks pass in both macOS and Linux Python environments. This article presents the four-plane, one-identity-spine architecture I believe deserves to be called a production flagship.

Figure 1: Original SVG cover. “Rate limit” in the title is deterministic fault injection, not a call to a real provider. The experiment validates control logic, not a provider’s SLA.

1. Background: Why a Chat Box Is Exposed the Moment It Enters Production

An AI demo usually follows three steps: a user types a sentence, the backend assembles a prompt, and a model returns text. The stage demonstration looks magical because the happy path is a straight road. Production is not a launch event. Users interrupt midway; providers rate-limit; tools time out; queues redeliver; permissions change; old memories expire; hostile instructions hide inside web pages or documents; and Finance asks why one turn consumed so many tokens.

Many teams respond by putting every new concern back into the prompt:

• Tell the model, in text, not to leak secrets.
• Ask it, in text, to confirm before dangerous actions.
• Tell it to switch to another model whenever something fails.
• Treat chat history as both task state and long-term memory.
• Guess from application logs how many times a tool really ran.

That is like asking an eloquent tour guide to become airport security, air-traffic control, maintenance, accounting, and the accident investigator at the same time. No matter how capable the guide is, one person should not hold every key. Natural language is excellent at expressing intent; deterministic systems are excellent at enforcing constraints. A production architecture gives each the job it can actually do.

2. Symptoms: Seven Illnesses That Appear When a Demo Becomes a System

1. Identity falls off. The model knows that “someone requested a restart” but not whether tenant, user, device, role, and session are verified.
2. Fallback becomes concealment. Every error switches providers, so even 401s, exhausted balances, and policy denials disappear behind more retries and more charges.
3. Approval is just chat prose. The model asks, “Are you sure?” An ambiguous “fine” becomes authorization, and that old authorization remains valid after parameters change.
4. Worker redelivery doubles writes. A tool succeeds, the process dies before storing COMPLETED, and the queue sends the task again. One restart becomes two; one transfer becomes two.
5. More memory becomes more danger. A service name retired three months ago is retrieved, and the model confidently acts on the wrong target.
6. There are many logs but no causal chain. The web tier, model gateway, and tool service each log something, but no trace ID answers who approved exactly what.
7. Success rate looks wonderful. A dangerous request correctly denied by policy counts as a failure, while a fluent answer that solves nothing counts as success.

Another system-prompt sentence cannot cure these. They belong to the platform.

3. Root Cause: Conversation State Is Mistaken for Business State

Chat messages can be truncated, summarized, reordered, or regenerated. A business task must have an explicit state, version, and owner. A message saying “executed” does not prove that a side effect happened exactly once. A model saying “the user approved” is not an immutable, argument-bound, unexpired approval record.

A minimal demo puts everything into one context window. A production platform does the opposite: separate experience, control, execution, data, and evidence into independently scalable, independently authorized, independently failing planes—then connect them with verified identity and tracing.

Figure 2: Original diagram. These boxes are divided by failure boundaries, not by team names. A Worker crash must not erase the conversation; a rate-limited model must not bypass policy; an expired memory must not pollute execution.

4. The Flagship Architecture: Four Planes and One Identity Spine

4.1 Experience Plane: Manage a Turn, Not Merely an HTTP Request

The experience plane contains adapters for Web, mobile, voice, messaging, and webhooks, plus a Turn Manager. It normalizes every channel into an envelope carrying verified identity, locale, device, attachments, deadline, and cancellation signal. It owns streaming, citations, reconnects, voice barge-in, and duplicate-message suppression.

Here is a child-friendly analogy. A teacher asks a question and a student starts answering. Halfway through, the teacher says, “The question changed.” A system that only understands HTTP hides the old speech bubble but lets the student continue computing and operating on the old question. A Turn Manager propagates cancellation through the model stream, retrieval, and tools. Cancellation is not a front-end animation; it is a context tree crossing every layer.

4.2 Control Plane: An AI Gateway Is More Than a Reverse Proxy

The control plane contains identity resolution, the AI Gateway, model routing, context assembly, policy, approval, task orchestration, and budgets. A traditional API Gateway mainly understands URLs, authentication, quotas, and protocols. An AI Gateway must additionally understand model capabilities, context windows, data classification, token ceilings, provider geography, streaming events, tool calls, and fallback eligibility.

A model may emit, “I suggest calling service.restart.” The Policy Engine independently evaluates role, tenant, resource scope, risk, and current approval, then returns ALLOW, DENY, or REQUIRE_APPROVAL. The model does not get the final vote.

4.3 Execution Plane: Remove Side Effects from the Chat Process

The execution plane contains a durable task queue, Workers, sandboxes, tool adapters, MCP Client/Server boundaries, leases, timeouts, concurrency controls, and idempotency records. Reading health and deleting data should not share an all-powerful Worker. High-risk tools belong in separate pools with short-lived credentials and narrowly allowed network egress.

Every side effect carries an idempotency key. Think of it as a parcel tracking number. If a network problem scans the same parcel twice, the warehouse still ships one box. Without the number, an at-least-once queue turns “delivered at least once” into “the action may happen many times.”

4.4 Data and Evidence Plane: Memory Is Not a Synonym for Vector Database

This plane stores conversation records, structured memory, retrieval documents, tasks, approvals, tool results, artifacts, audit events, metrics, traces, evaluations, and cost. Different data needs different retention and access. A user preference may live for years; raw voice may disappear in days; an approval event may require regulated retention; a secret should enter neither prompt nor ordinary log.

Structured memory needs at least source, scope, version, expires_at, sensitivity, and deletion semantics. Vector similarity answers “does this look related?” It cannot answer “is this still true?”

4.5 Identity Spine: Never Drop the Passport at a Handoff

Tenant, user, role, device, session, and delegation chain must propagate from entry to routing, policy, task, tool, and audit. Putting a username inside the prompt is not identity propagation. Authorization needs verified structured claims that a model cannot rewrite.

The lab’s read-only happy path prints that chain:

Figure 3: Real lab-output capture. A synthetic identity passes policy, routes to a local simulated provider, receives authorization for a read-only tool, sanitizes the result, and reaches COMPLETED. No real model or tool is called.

5. What Actually Happens During One Request?

Take “check the storage appliance’s health and recommend the next step” as an example. A request should cross these accountable handoffs:

1. The Channel Adapter verifies the channel signature and creates a common envelope.
2. The Turn Manager establishes the turn, deadline, and cancellation tree.
3. The Context Assembler selects only data valid for this tenant and purpose.
4. The AI Gateway chooses a model by capability, data class, latency, and cost.
5. The model proposes a plan or tool intent; it never receives broad production credentials.
6. The Policy Engine rechecks tool name, normalized arguments, scope, and risk.
7. If approval is required, the durable task enters WAITING_APPROVAL and the UI shows the exact impact.
8. A Worker takes a lease, runs in a sandbox with short-lived credentials, and writes an idempotent result.
9. A sanitizer filters tool output, and the Turn Manager streams the response.
10. Trace, audit, tokens, cost, and business outcome are archived under one request identity.

Figure 4: Original diagram. Air travel separates ticketing, security, the gate, the crew, and baggage tracking. Handing every role to an “eloquent captain” is not smarter; it makes incidents impossible to contain.

The central rule is: propagate structured context at every hop, and reauthorize every side effect. Entry authentication does not make arbitrary model-generated arguments trustworthy.

6. Failure One: The Model Returns 429—What May Fall Back, and What Must Stop?

For a 429 or a transient 5xx, a platform may switch to a compatible model within a bounded retry budget. A 401, invalid credential, exhausted entitlement, or compliance denial should expose the root cause immediately. Treating every error as fallback turns a configuration failure into an expensive, slow ghost incident.

Figure 5: Real lab-output capture. The primary route returns synthetic 429, fallback returns 200, retry budget is one, and the reason is recorded. Authentication failures are explicitly ineligible.

Figure 6: Original diagram. Routing considers more than availability: may the data leave the region, does the model support tools or structured output, how much budget remains, and is there enough deadline left?

An auditable routing decision records the candidate set, exclusion reasons, final route, model and prompt versions, attempt count, each error type, tokens, and cost. Otherwise “the system changed models” becomes an untestable explanation for every quality shift.

7. Failure Two: The Approval Gate Lives Outside the Model

A dangerous write cannot rely on a model asking, “Are you sure?” An approval record must bind the exact tool, canonical arguments, resource scope, requester, approver, expiry, and one-time nonce. If the plan’s arguments change, the approval becomes invalid.

Figure 7: Real lab-output capture. Even a synthetic platform-admin identity receives REQUIRE_APPROVAL. The task persists in a waiting state and the tool is not called. A powerful role is not an approval bypass.

Figure 8: Original diagram. A production matrix also considers tenant, resource tags, time window, change calendar, risk score, and separation of duties. The model submits a request; deterministic policy stamps it.

The approval interface must not show only a model summary such as “optimize the service.” It should display the actual argument diff, number of affected objects, rollback action, evidence links, and expiration time. A human approves a precise work order, not a blank permission slip.

8. Failure Three: Prompt Injection Cannot Be Solved by Prompt Alone

Prompt injection may come directly from a user or hide inside a web page, email, PDF, code comment, or tool result. It is untrusted data pretending to be a control instruction. Writing “do not obey bad people” in a system prompt is like putting a “bad people may not enter” sign at a school gate. It has reminder value, but it is not access control.

Figure 9: Real lab-output capture. The synthetic input combines instruction override with secret-exfiltration intent. Policy returns DENY; no model or tool is called and no secret is logged. The lab’s keyword policy demonstrates a control point—it is not a production-grade injection defense.

Production defense is layered: label source and trust level; place retrieved text inside an explicit data boundary; allowlist tools and validate arguments against schemas; separate read and write credentials; approve high-risk actions; restrict network egress; inspect input, plan, tool result, and final output separately; and continuously evaluate against red-team samples. Even if the model is manipulated into saying the wrong thing, policy and execution should keep it from doing the wrong thing.

9. Failure Four: The Worker Crashes in the Most Dangerous Millisecond

The hardest distributed-systems moment is after a tool has produced a side effect but before the Worker stores COMPLETED. The queue sees only an expired lease and redelivers the task. Without an idempotency record, the next Worker performs the action again.

Figure 10: Real lab-output capture. The first attempt creates one synthetic effect and crashes. Recovery resumes from the before-tool checkpoint, the second call finds the idempotency record, final effect count remains one, and the task completes.

Figure 11: Original diagram. The chat window may close while the durable task keeps its state. A production machine may also need CANCEL_REQUESTED, DENIED, FAILED, and COMPENSATING, with explicit transition authority and timeout rules.

Derive the idempotency key from tenant, business task, tool, canonical arguments, and version, then commit it atomically with the result. When an external system is not naturally idempotent, use its request-key feature, a business unique constraint, or a compensation workflow. An already_done = true variable in Worker memory disappears with the process and solves nothing.

10. Failure Five: Stale Memory Is More Dangerous Than No Memory

Forgetting a preference is annoying; confidently remembering a retired service, old contact, or expired permission is an incident. A memory system must answer who said it, whom it applies to, which version it is, when it expires, whether it can be deleted, and why it was retrieved.

Figure 12: Real lab-output capture. The current item carries source and version, one stale item is counted but never enters the prompt, and memory is explicitly deletable.

Figure 13: Original diagram. Memory is a library loan, not a tattoo: it needs a source card, borrowing scope, and return date. The vector store finds similar books; policy decides whether the book may be borrowed now.

Useful categories include session working memory, user-confirmed long-term preference, low-confidence behavioral inference, organizational knowledge, and task artifacts. An inference must not silently become fact. A user correction should create a new version and suppress the old one. Memories about people, permissions, and production resources deserve shorter lifetimes and stricter verification.

11. Observability: Trace the Decision, Not Only the Final Answer

AI request latency is not one number. TTFT answers “how long until it started speaking.” Full response time answers “how long until it stopped speaking.” E2E for a tool task answers “how long until the work was actually complete.” It is like food delivery: fast order acceptance does not mean fast delivery, and an app saying “delivered” does not prove the correct meal reached your hands.

One trace should join entry, retrieval, every model attempt, policy decision, approval wait, tool execution, retry, streaming response, and final business outcome. Tokens, cost, and model version are span attributes. Prompt and tool content should be redacted or hashed according to data class; debugging must not create a new secret warehouse.

Figure 14: Real lab-output capture. The fault lab joins ai.request, gen_ai.chat, and tool.execute in one waterfall. Durations are tiny because all behavior is local and synthetic; they are not real-model latency claims.

Figure 15: Original diagram. An operating panel must answer: is it fast, is it affordable, did it solve the task, and did it stay authorized? Optimizing only tokens or latency pushes the system in the wrong direction.

Organize metrics into four groups:

• Experience: TTFT, E2E, cancellation propagation, stream interruption, and user retry rate.
• Inference: model errors, fallback, context size, input/output tokens, and structured-output validation failures.
• Action: tool success, approval wait, duplicate effects, timeouts, and compensation success.
• Outcome and risk: resolution rate, correct-denial rate, human takeover, cost per successful outcome, and blocked injection or authorization attempts.

A correct denial is a safety success, not a business failure. A polished paragraph that completes no task is not success merely because HTTP returned 200.

12. Where MCP Fits: A Protocol Boundary Is Not an Authorization Boundary

MCP standardizes how Hosts, Clients, and Servers expose tools, resources, and prompts, dramatically lowering integration cost. But “the tool is discoverable” does not mean “this caller is authorized to run it.” A Server describes capability; the platform still enforces tenant isolation, argument validation, risk classification, approval, and network policy before every call.

Think of MCP as a standardized electrical outlet. Once the connector is common, both a lamp and a power drill can plug in. The outlet cannot decide whether a child may use the drill. Permission, circuit protection, and the work permit remain the building’s job.

High-risk Servers deserve separate processes or containers, minimum credentials, and fixed egress. Do not combine filesystem access, shell, production control, and personal data into a universal MCP Server. Version the tool catalog too: the schema the model sees, the schema policy validates, and the implementation the Worker runs must agree.

13. Deployment Shape: Scale Stateless Edges and Protect Stateful Cores

Figure 16: Original diagram. Horizontal scaling does not mean copying every box three times. The critical properties are one task lease holder, one idempotency record per effect, and a trusted source of truth for approvals and audit.

A practical deployment keeps Channel, Turn, Gateway, Policy, and task APIs stateless and replicated. Tasks, approvals, and idempotency keys enter a transactional database. A queue manages leases and redelivery. Workers are split by tool risk and sandboxed. Large artifacts use object storage. Memory and retrieval stores enforce tenant filters and carry version metadata. OpenTelemetry Collectors gather traces, metrics, and logs, while audit events flow to stricter immutable storage.

An early team does not need twenty microservices. The four planes are responsibility boundaries and can begin inside a modular monolith. Split processes when scaling, authority, or failure domains truly diverge. Prematurely distributing everything to look “flagship” buys network timeouts and consistency problems before it buys value.

14. From Demo to Platform: A Realistic 90-Day Path

Days 1–30: install brakes first. Standardize the identity envelope; create tool allowlists and JSON Schemas; split read from write credentials; add high-risk approval; propagate one trace ID; record routing decisions; and establish a minimum red-team set. Connect fewer tools rather than give the model one universal token.

Days 31–60: separate tasks from conversation. Add a durable state machine, leases, timeouts, and idempotency; propagate cancellation; give structured memory source, version, and TTL; classify retryable errors; define success by outcome instead of HTTP code.

Days 61–90: operate by evidence. Build golden tasks and offline evaluations; use shadowing and canaries; analyze cost and quality by tenant, model, and tool; rehearse provider 429, queue backlog, Worker crash, approval timeout, unavailable memory store, and audit-pipeline failure; turn recovery results into release gates.

Add components only when a clear pain requires them. A ten-person internal assistant and a multi-tenant enterprise platform can share principles without copying the same deployment scale.

15. Reproducible Fault Lab: What the Six PASS Results Prove

The downloadable Python fault-injection script uses only standard-library SQLite, hashing, JSON, and timing. It makes no network request, calls no real model or tool, and loads no secret. On Windows 11:

py -3 ai_platform_fault_lab.py --clean

On Ubuntu 26.04 and macOS 26:

python3 ai_platform_fault_lab.py --clean

The lab tests control boundaries, not model quality, production security certification, or throughput. Read the README, or inspect the structured traces.jsonl and metrics.json.

Figure 17: Real lab-output capture. The identity-bound read, bounded 429 fallback, high-risk approval, pre-execution injection denial, crash-resume deduplication, and stale-memory exclusion all pass. Cost is zero because there is no external call. A 60% successful_requests rate excludes one correct denial and one approval wait, proving that a single “success rate” needs state-aware interpretation.

The same script passes on two platform families with different Python versions. Trace durations differ, as expected. A production implementation must replace the toy keyword policy with layered controls, replace local SQLite with reliable shared state, and replace simulated tools with minimum-authority adapters.

16. Q&A

Q1: If I use the strongest model, can I build fewer platform controls?

No. A better model can reduce some reasoning errors; it cannot provide transactions, verified identity, approval, idempotency, or audit. A faster car needs reliable brakes more—not fewer.

Q2: What is the exact difference between an AI Gateway and an API Gateway?

A traditional gateway handles routing, authentication, quotas, and protocol concerns. An AI Gateway additionally handles model capability and version, context windows, data residency, token and cost budgets, streaming events, tool calls, fallback eligibility, and model-level observability. The layers may share infrastructure, but their responsibilities differ.

Q3: Why not ask a model to detect prompt injection?

A model can contribute a signal but cannot be the only judge. Attack content may resemble legitimate documents, and the detector itself reads the hostile context. Deterministic allowlists, schemas, authorization, sandboxing, egress limits, and human approval provide boundaries outside the model.

Q4: Does every tool call require human approval?

No. Read-only, low-risk, reversible, narrowly scoped actions can be policy-approved automatically. Dangerous, irreversible, cross-tenant, or wide-impact actions need approval. Approving everything creates fatigue and blind clicking, so risk classification matters.

Q5: Does exactly-once eliminate duplicate side effects?

End-to-end exactly-once is difficult to promise. A realistic design uses at-least-once delivery, idempotent execution, uniqueness constraints, auditable results, and compensation or reconciliation for non-idempotent actions. The lab proves that redelivery does not repeat its synthetic effect.

Q6: How do RAG, long-term memory, and chat history differ?

Chat history records what happened in the current conversation. RAG retrieves material from an external knowledge source for the current question. Long-term memory stores cross-session information with source and lifecycle. All may enter context, but their permission, retention, and confidence differ.

Q7: If an MCP Server declares permissions, must the platform check again?

Yes. Capability metadata cannot replace caller identity and policy, and a remote Server may change or be compromised. Pin versions, restrict tools, validate schemas, limit egress and timeout, and reauthorize each high-risk call.

Q8: Must an AI Platform use microservices, Kubernetes, and a vector database?

No. A flagship starts with complete responsibility and evidence, not component count. A modular monolith, transactional database, and reliable queue can implement the four-plane boundaries. Split only when workload, organization, or compliance demands it.

Q9: How do I measure whether the platform is useful?

Measure outcomes: first-attempt resolution, human time saved, correct denial, incidents, cost per successful task, and repeat usage. Model leaderboards, token count, or text-similarity scores explain pieces but do not prove work completed.

Q10: Which three elements should be implemented first?

The identity envelope, deterministic policy before tools, and one trace ID across request, model, and tool. They answer “who is acting,” “may this happen,” and “what happened”—the foundation for approval, idempotency, memory, and evaluation.

Continue Reading: Five Production Deep Dives

This article defines the complete boundary. The five focused articles below open the control points most likely to fail and attach reproducible fault injection and acceptance evidence. Return to the AI Platform Architecture Practice Reading Map for the complete navigation.

• Why One Task Runs Twice: Worker Crashes, Idempotency, and Durable Execution
• You Said Stop—Why Is the AI Still Running? Turn Manager, Cancellation, and Barge-in
• A 429 Does Not Mean “Switch Models”: AI Gateway Routing, Fallback, and Circuit Breaking
• AI Remembering You Is Not Always Good: The Structured-Memory Lifecycle
• Stop Testing Only the Answer: Production Release Gates for an AI Platform

17. References and the Architectural Bottom Line

• NIST AI Risk Management Framework
• OWASP Top 10 for LLM / GenAI Applications
• Model Context Protocol: Architecture
• OpenTelemetry Generative AI Semantic Conventions
• HumanLayer: 12-Factor Agents

This architecture may begin with fewer components and a modular monolith, then evolve with scale. One boundary cannot be negotiated away: the model must never be the intent interpreter, permission approver, side-effect executor, and accident auditor at the same time. The model proposes; the platform owns the consequence.