Temporal | Marcin Wylot, PhD

Incremental Map-Reduce LLM Pipeline with Idempotent State and Prompt-Injection Defense

Thu, 09 Jul 2026 00:00:00 +0000

For the same leadership-development AI platform I built the production infrastructure for ( ), I designed and built the insight-synthesis layer — the pipeline that reads each user’s entire conversation history and distills it into a small set of durable, personalized insights that evolve as they keep using the product. The system is an incremental map-reduce over an LLM: a MAP stage extracts a structured signal per conversation, a REDUCE stage folds those signals into insights against a bounded, durable profile of the user’s history. Built on Temporal for reliability, it re-processes only what changed (keeping per-run LLM cost proportional to new activity rather than total history), treats every LLM output as untrusted input, and is idempotent under retries — no false success, no double-counted history, no silently dropped conversations.

The Challenge

The platform runs a Socratic AI companion whose value compounds over time — the more a user reflects, the more the product should understand their evolving practice. Turning that accumulated history into trustworthy, longitudinal insight raised a set of problems that single-shot LLM calls don’t have:

Longitudinal, not per-session: insights must reflect a user’s whole arc — dozens to hundreds of conversations — not one chat
Unbounded, ever-growing history: a user’s transcript only grows, so naively re-reading everything on each run is cost-prohibitive at a per-user, minutes-level cadence
Cost discipline: only conversations that actually changed should ever hit the LLM again
Correctness under a distributed retry engine: a retried or duplicated run must not double-count history, advance state past unprocessed data, or persist a partial result as “done”
LLM output is untrusted: extracted content is persisted and fed back into later prompts — out-of-vocabulary or injection-poisoned output must never corrupt durable state
Prompt injection: conversation text is fully user-controlled and flows directly into extraction prompts
Never truncate: an arbitrarily long conversation must be fully processed, not clipped to fit a context window

They needed:

Personalized, evolving insights synthesized from a user’s complete conversation history
Cost that scales with new activity, not with total history size
Provable correctness under Temporal’s at-least-once activity execution
A hardened trust boundary around every LLM call whose output touches persisted state
Full, lossless processing of long conversations

Solution Architecture

I designed a two-stage map-reduce driven by a completion poller, with a durable per-user profile that lets REDUCE fold new activity forward instead of re-reading history:

graph TD poller[5-min completion poller] --> map[MAP: extract changed conversations] map -->|conversations remain| drain[Drain: mark 'draining', re-fire next cycle] map -->|coverage below threshold| retry[Skip REDUCE: re-MAP next cycle] map -->|extract set complete| profile{Profile current?} profile -->|dirty / schema bump / first run| bootstrap[Bootstrap: hierarchical fold over all history] profile -->|current| foldfwd[Fold-forward: fold only aged-out extracts] bootstrap --> hash{generation_hash changed?} foldfwd --> hash hash -->|no| noop[No-op: idempotent skip] hash -->|yes| synth[Two-call synthesis: pattern + gated shift] synth --> store[(Persisted insights)]

MAP — per-conversation structured extraction

The MAP activity extracts one structured ConversationExtract per conversation that needs (re)processing:

Change detection: a per-conversation source_fingerprint is compared against the persisted extract, so only genuinely changed conversations are re-mapped — every unchanged conversation is skipped before any token is spent
Bounded, isolated parallelism: candidate conversations are mapped concurrently under an asyncio.Semaphore, with Temporal heartbeats as they complete; a failure on one conversation is isolated and never fails the user’s run
No-truncation windowing: oversized conversations are split into token-budgeted windows, each extracted separately and then merged — content is never clipped, and the merge deliberately preserves the earliest framing markers as frozen “then” anchors
Model-facing schema with code-owned provenance: extraction is enforced as a Pydantic model via tool-calling structured output; factual provenance the model can’t know (conversation UUID, timestamps) is injected by code, not produced by the model
Cold-start draining: when more conversations remain than one batch should map, the user’s row is set to draining and the workflow returns in_progress so the poller re-fires next cycle — bounded batches over poll cycles instead of an unbounded continue_as_new
Fail-closed coverage gate: if the per-batch error ratio exceeds a configured threshold, REDUCE is skipped entirely and the failed conversations re-map next cycle — the system never synthesizes insights from partial coverage

REDUCE — folding extracts into insights

The REDUCE activity aggregates a user’s extracts into the insight blob, backed by a durable PracticeProfile that keeps the token budget bounded no matter how long the history grows:

Bootstrap vs. fold-forward: on first run, a schema bump, or an invalidated profile, a chronological hierarchical reduce folds all history into a compacted profile (running outline + principle/dimension tallies + frozen earliest anchors). On every subsequent run it folds forward — advancing a covered_through watermark and folding only the extracts that have aged out of the granular recent window
Recent-at-full-granularity, history-compacted: the most recent extracts are read verbatim (they drive the current pattern); everything older lives in the bounded profile outline. This is what caps cost as history grows without number
Idempotency, three ways: a SHA-256 generation_hash over the inputs short-circuits any run whose inputs are unchanged; the fold happens before the watermark is mutated, so a failure can never advance covered_through past an unsummarized batch; and the insight write plus its side effects commit in a single caller-owned transaction
Two-call gated synthesis: a pattern unit (through-line + unexplored-principle) and a gated “what’s shifted” unit run concurrently with per-call failure isolation; the expensive shift analysis is gated behind minimum-history and minimum-timespan thresholds
Sharpen, don’t whiplash: prior observations are fed back into the synthesis prompt so insights evolve incrementally rather than lurching between runs, with section-level freshness resolving fresh output against the previous result

AI Security and Trust Boundaries

Because extracted output is persisted and re-fed into later prompts, I treated the LLM as an untrusted component and hardened the boundary on both sides of every call:

User content is DATA, not instructions: transcript text has its angle brackets escaped before entering the prompt, so a message can’t close the <transcript> wrapper or inject markup the model would treat as instructions — the transcript stays human-readable while being inert
LLM output is allow-listed before persistence: every principle and behavioral dimension the model emits is checked against the controlled catalog/enum and dropped if it isn’t a member, so a prompt-injection attempt can’t poison durable state with fabricated vocabulary
Provenance is code-owned: conversation identity and timestamps are injected by code and never taken from model output, so the model cannot misattribute an extract to another conversation or user
Fail-closed everywhere: partial MAP coverage, a failed profile fold, or a failed core synthesis all refuse to persist a “success” — the run is marked for retry with prior state left intact, rather than shipping a confidently-wrong result

Technical Implementation

Temporal orchestration: MAP and REDUCE are activities with explicit start-to-close (10 min) and heartbeat (5 min) timeouts and bounded retry policies; the workflow is triggered per-user by a completion poller, and REDUCE re-raises retryable failures so Temporal — not a swallowed exception — owns the retry
Structured output via tool-calling: every LLM interaction (extraction, profile fold, pattern synthesis, shift synthesis) is a schema-validated tool call into a Pydantic model, eliminating free-text parsing
Langfuse observability: one trace per conversation in MAP (session-scoped to the conversation, each window a separate generation) and one per-user trace for the whole REDUCE, giving prompt/latency/cost visibility at both granularities
Pure-function core: partitioning, window/epoch merging, tally accumulation, and the generation-hash preimage are all pure functions, unit-tested independently of the database and the LLM — the deterministic logic is provable without touching a model
Config-driven budgets: token budgets, concurrency limits, batch sizes, the coverage error-ratio threshold, and the shift-gating thresholds are all externalized settings rather than hard-coded constants

Key Design Decisions

Incremental fold-forward over full re-synthesis:

Re-reading a user’s entire history on every run is simple but scales cost with total history — untenable at a minutes-level cadence
The durable profile + watermark makes per-run cost proportional to new conversations, so a long-tenured user costs no more per run than a new one

Watermark advanced only after a successful fold:

Advancing covered_through before persisting the fold would let a mid-run failure silently drop the unsummarized batch from history forever
Folding first and mutating second makes the whole REDUCE safely retryable — a post-commit retry sees the advanced watermark and correctly folds nothing

Treat LLM output as untrusted input:

Most pipelines validate the shape of structured output; here the contents are allow-listed against a controlled vocabulary before they ever reach durable state
Combined with transcript escaping and code-owned provenance, this closes the prompt-injection path into persisted, re-prompted data

Cold-start drain over continue_as_new:

Bounded per-cycle batches that lean on the existing poller are simpler to reason about and observe than a self-continuing workflow, with no loss of eventual completeness

Fail-closed synthesis:

A coverage gate and retryable-error contract guarantee the system would rather produce nothing this cycle than a plausible insight synthesized from partial or corrupted inputs

Results & Impact

Delivered a longitudinal insight-synthesis layer that distills a user’s full conversation history into three evolving, personalized insights (a through-line pattern, an unexplored principle, and how their framing has shifted over time)
Architected an incremental map-reduce that keeps per-run LLM cost proportional to changed conversations rather than total history, via per-conversation fingerprint change-detection and a durable fold-forward profile
Made the pipeline idempotent under Temporal’s at-least-once execution — generation-hash short-circuiting, fold-before-mutate watermarking, and single-transaction side effects eliminate false success and double-counting
Hardened the LLM trust boundary end-to-end: prompt-injection escaping of user content, controlled-vocabulary allow-listing of persisted model output, and code-owned provenance
Guaranteed lossless processing of arbitrarily long conversations through token-budgeted windowing that splits, extracts, and merges without truncation
Instrumented full Langfuse observability at both conversation and user granularity for prompt, latency, and cost tracing
Isolated all deterministic logic into a pure-function core with dedicated unit tests, keeping correctness verifiable independently of the LLM

Technologies

Python, Temporal, Structured Output (tool-calling), Pydantic, asyncio, Langfuse, PostgreSQL, SQLAlchemy, SHA-256 idempotency, Prompt-Injection Hardening

Your Extraction POC Works on 10 Documents. Here's Why It Dies on 10,000.

Thu, 09 Jul 2026 00:00:00 +0000

I build the extraction pipelines that have to hold up on the whole corpus, not the slice in the demo. Which usually means I get called in after the celebration. The vendor demo ran clean: a dozen hand-picked PDFs went in, structured fields came out, the fields were correct, and everyone in the room exhaled. Somebody said the word “magic.” A six-figure number got attached to a roadmap. Then the pipeline met the real corpus — the ten thousand documents the business actually owns, the ones nobody curated — and the output quietly fell apart. Not with an error. With confident, plausible, wrong results that nobody caught until they were already downstream.

Here is what that demo actually proved: the model can do the task once, on inputs chosen to make it look good. That is a real fact, and it is almost never the fact in question. The model can extract the field. Modern models are genuinely good at reading a document and returning structure. What the demo proved nothing about is the only question that matters at scale: will this be true, cheaply, every single time, when part of the batch fails and the whole thing gets retried at two in the morning? “Can the model do this” is a research question, and it’s usually settled before you walk in the door. “Will this hold, at cost, under failure, across the full ugly corpus” is the entire job. Those are different disciplines. Confusing them is what turns a signed-off POC into a write-off.

The gap is engineering, not intelligence

When the pipeline cracks, the instinct in the room is always the same: reach for a smarter model. Swap the mid-tier model for the frontier one, add another reasoning step, wait for next quarter’s release. I understand the reflex — the failure feels like the model isn’t smart enough. It almost never is.

The gap between a working demo and a working system is four engineering failures, and not one of them is fixed by a model upgrade. Cost that scales with the wrong thing. Silent truncation. Confident but partial coverage. And no correctness when the system retries itself. A better model runs each of these failure modes more eloquently; it does not remove any of them. That’s the part that decides whether the six figures land, and it’s the part that’s invisible in a demo. These four are predictable. Someone who has actually shipped extraction at scale can look at your corpus and your cadence and tell you which one bites you first — before you write the check, not after.

Let me take them one at a time. Each has a fix, and the fix is something you build, not something you buy.

Failure one: your cost scales with the wrong thing

On ten documents, cost is a rounding error. You will never notice it, which is exactly why the demo hides it. On ten thousand documents reprocessed at real cadence — every hour, every few minutes, forever — cost is the line item that quietly ends the project. Someone in finance eventually asks why the inference bill looks like a second headcount, and there is no good answer, because the pipeline was built to re-read the entire world on every run.

That’s the trap: the naive design’s cost scales with your total state, not with what changed. It reprocesses everything because it has no idea what’s new.

I built a pipeline for a US enterprise’s public-facing Socratic AI companion — a system that reads each user’s entire conversation history and distills a few durable, evolving personal insights. It runs continuously, per user, on a five-minute Temporal schedule. The naive version of this is a cost catastrophe: re-read every conversation for every user every five minutes, forever. So it doesn’t. Before a single token is spent, each conversation is fingerprinted — a SHA-256 hash over its message count and its latest timestamps — and any conversation that hasn’t changed since the last run is skipped outright. The extraction stage only ever touches conversations that actually moved. On top of that, the whole synthesis step is short-circuited by a second hash: if none of the relevant inputs changed, there is no model call at all. And the profile it maintains is deliberately bounded — it does not grow with how long someone has been a user.

I’m not going to quote you a cost ratio, because the honest answer is that the savings depend on your traffic. What I will tell you is that this is a design property, not a tuning parameter. Cost proportional to change is something you architect in at the start. You cannot bolt it onto a pipeline that was built to re-read the world, and you certainly cannot buy it from a better model.

Failure two: silent truncation, the bug you never see

This is the failure I most want to catch before anything ships — and the one a demo is structurally unable to show you.

Every other failure at least announces itself eventually — a bill, a crash, an empty result. Silent truncation doesn’t. It produces a confident, well-formed, completely reviewable answer derived from part of the input. A real document — a long conversation, a dense report, a full transcript — doesn’t fit the model’s context window. The naive pipeline clips it to fit. No error is raised. No warning is logged. The model dutifully extracts structure from the fraction that happened to fit — everything past the cutoff silently gone — and returns it looking exactly like a correct result. It passes human review, because a reviewer checks whether the output is plausible, not whether the input was complete. You cannot see what was silently dropped.

At ten documents, none of them overflow, so this bug is invisible in the demo by construction. At ten thousand, the long ones always exist, and they are usually your most important records.

The fix is not subtle, but it has to be deliberate. In the companion pipeline, an oversized conversation is split into sixty-thousand-token windows using a deterministic size estimate, each window is extracted independently, and the results are merged losslessly. The rule that matters is the one about failure: all windows are required. If any single window fails to extract, the entire conversation is marked as errored and nothing partial is ever persisted. There is no code path that writes half a document and calls it done. That’s what “no truncation” actually means as an engineering commitment — not “we made the window bigger,” but “the system structurally cannot ship a fraction of a document as if it were whole.”

Failure three: confident, and wrong about half of it

Here is a fact about running anything at scale: some fraction of every batch fails. A model call times out, a document is malformed, a rate limit trips. On ten inputs you can pretend this away. On ten thousand, failure isn’t an exception, it’s a rate.

The dangerous move — the one I see constantly — is to treat partial success as success: take whatever came back from the batch, synthesize a result from it, and ship it labeled “complete.” Now you have a confident summary that silently omits whatever failed, and nobody downstream knows a piece is missing. That’s not a smaller version of the right answer. It’s a wrong answer wearing the right answer’s clothes.

The pipeline I built fails closed instead, and it does so at two levels. At the coarse level there’s an explicit coverage gate: if the failure rate in a batch exceeds twenty percent, the synthesis stage does not run at all — the run returns “in progress” and the failed conversations get re-processed on the next cycle. But the gate is the blunt guard, not the real guarantee. The guarantee is structural: no conversation is ever recorded as covered until its extract has actually been folded in. A watermark tracks how much history has genuinely been synthesized, and it advances only over what succeeded — so a conversation that failed this cycle is never quietly marked done. It simply comes back next cycle. The system would rather tell you “not finished yet” than hand you a confident result that’s missing a piece it can’t name. That is a product decision as much as an engineering one, and it’s the kind of decision that only gets made after someone has been burned by the alternative. Correct under failure is a posture you choose on purpose.

Failure four: no correctness when things retry

Production extraction doesn’t run as a script on your laptop. It runs on distributed execution engines, and those engines retry. This is not a bug in them — it’s the contract. Temporal, which is what I use for durable orchestration, executes activities at least once: a step that already completed can be run a second time after a worker crash or a timeout, because the engine couldn’t confirm the first attempt finished. At-least-once is the price of durability. It means every step in your pipeline must assume it may execute twice.

A pipeline that ignores this double-counts. It folds the same batch of extractions in twice, or it marks a half-finished run as complete because the retry landed on the wrong side of a state change. These bugs are miserable to reproduce because they only appear under exactly the failure timing the demo never experiences.

So idempotency isn’t a nice-to-have here, it’s load-bearing, and in the companion pipeline it’s enforced three ways. That “covered-through” watermark is advanced only after a fold has successfully completed — the fold runs and is allowed to fail before the watermark ever moves, so a crash can never skip a batch by advancing state past work that didn’t happen. A generation hash over the run’s actual inputs short-circuits any run whose inputs haven’t changed, so a spurious retry does no synthesis at all. And retryable failures are deliberately re-raised so that Temporal owns the retry, rather than being swallowed and papered over. The orchestration engine is configured to match: bounded automatic retries, heartbeats on the long-running steps so a stalled worker is detected instead of hanging, and — deliberately — a poll-driven drain in bounded per-cycle batches rather than a cleverer self-continuing construct, because a system you can reason about and observe beats a system that’s elegant and opaque.

One more property rides on the same discipline. This pipeline persists a profile and feeds it back into the next run’s prompt, so a single poisoned extract could otherwise steer every future cycle. Nothing reaches persisted state unvetted: model output is allow-listed before it’s stored — emitted principles checked against a controlled catalog, behavioral dimensions normalized against a fixed vocabulary, anything out-of-vocabulary dropped. The state that gets re-prompted is state the system already vouched for. Correct under retry and secure by design turn out to be the same instinct — never trust output you didn’t verify, whether the threat is a double execution or a prompt injection — and both are things you can point to in the code, not things you hope for.

Scale is a discipline, not a trick

If all of this only ever showed up in one pipeline, you could dismiss it as one clever hack for one weird problem. It isn’t. It’s the same discipline, and I watch it recur across domains that have nothing to do with each other.

For a sports-media company generating automated broadcast narratives, the pipeline processes hundreds of games a season across nine narrative categories each. Same problem shape, different clothes. Cost discipline shows up as prompt caching: one large system prompt — rosters, eighty-seven stat types, the full speech-to-text transcript, tens of kilobytes of it — is cached once and reused across all nine parallel category calls for a game, so only the first call pays full freight and the rest hit cache reads. Coverage discipline shows up as blast-radius control: failure is isolated per category, per element, per game, so one empty category never sinks the broadcast. And correctness shows up as an independent judge model that verifies every extracted element before it survives to the final curated set. I deliberately tore out an earlier framework-heavy design and rebuilt it on plain async orchestration for direct control over exactly these concerns.

For a civil-engineering firm generating site-assessment reports, hundreds of data points per report flow through an async work queue with an explicit state machine — pending, processing, done, failed — tracked in a real database. It is the least glamorous thing I could describe to you, and it is exactly the thing that turns “it worked when I ran it” into “I can tell you precisely which of the N items succeeded, which failed, and why.” Every generated section carries provenance back to a source file and page number, and the whole thing runs through human verification checkpoints. The unglamorous state machine is the product.

Three industries, one shape. That’s how you know it’s a practice and not a party trick.

Why a POC has to predict production

The reason the demo lies is now precise: it exercises none of the four failure modes. It runs short, curated, low-failure, single-pass inputs, which is to say it runs the exact conditions under which all four failures are invisible. Of course it looked like magic. It was staged in the one environment where the hard problems don’t exist.

A POC worth the name does the opposite on purpose. It deliberately runs the ugliest, longest, highest-failure inputs in the corpus — the documents that overflow the window, the batches with a bad failure rate, the load that makes the retry timing actually happen — and it reports back on the four things that predict production: what this costs at real cadence, what happens to the documents too big to fit, what the system does when a fifth of the batch fails, and whether it stays correct when every step runs twice. That’s the thing I build first, and it’s the differentiator worth paying for: a proof of concept that tells you the expensive truth cheaply and early, while the number on the roadmap is still an estimate and not a sunk cost. A POC that de-risks the six-figure bet before you place it is a fundamentally different artifact from a demo that makes the bet feel safe.

Each of these is its own way to die

I’ve compressed four failure modes into one essay, and each of them deserves its own. Trusting an extracted value as if it were a fact — when to gate, verify, and refuse — is a whole discipline. The data backbone beneath production AI, the state machines and provenance that make the intelligence trustworthy, is the depth that everything else stands on. Knowing when a problem wants a deterministic pipeline versus an agentic one — and refusing to reach for autonomy where control is cheaper and safer — is its own decision with its own failure modes. And what an agentic system actually costs to run, once it’s making its own choices, is a question most teams don’t ask until the bill arrives. Each gets its own full treatment. Consider this the map.

Before more money goes in

So here’s the honest question. If your extraction POC only ever ran on the easy documents — the curated dozen, the ones that fit, the batch that happened not to fail — and nobody has yet pointed it at the full, ugly corpus at real cadence, then you don’t actually know what you have. You have a demonstration that the model can do the task. You do not yet have evidence about cost, truncation, coverage, or correctness under retry. That gap is precisely the thing worth closing before the next tranche of budget goes in.

I’m not selling you a rescue. I’m telling you it’s cheaper to find out what breaks in a POC designed to break it than to find out in production, in front of your boss, with the six figures already spent. If that’s the conversation you’d rather have now than later, that’s the one I’m good at.