The starting point
An AI capability becomes part of a software system when people depend on its state, access rules, and failure behavior. The model may generate a plan or extract a statement, but the application still decides who can request the work, which information it may use, and whether an output is ready to act on.
Choose an architecture from the workload outward. Record expected input size, request rate, processing duration, the cost of repeating a task, and the consequence of an incomplete result. A short answer in a support screen has different needs from document extraction or a generation job that produces multiple assets.
The useful design is the one whose behavior stays understandable under interruption. Work through a browser closing, a provider timing out, and a worker restarting. For each case, identify the authoritative record, the recovery action, and what the person will see.
The decisions that shape the implementation
Use a synchronous call when the interaction can finish within its budget
A direct request can fit a bounded classification or a short response with predictable input sizes. Set a deadline, expose a useful failure state, and keep an external write separate from the generated suggestion. Measure the slow requests as well as the average: a small group of long calls can make an otherwise simple interface difficult to use.
Use durable jobs when the work needs its own lifecycle
Long-running generation, extraction, and asset processing benefit from an acknowledged job that survives the browser session. Store the request and state durably, buffer work in a queue where appropriate, and make retries bounded. A worker lease identifies the current owner; a completion write must still match that ownership so a late worker cannot overwrite a newer result.
Treat retrieval as a source-access boundary
When an answer depends on an existing document collection, retrieve material the requesting person is permitted to use before calling the model. Keep source identifiers and versions with the result. Retrieval can supply relevant context; it does not make a generated interpretation correct or grant access to another workspace’s records.
Separate responsibilities before separating every service
A modular application can own accounts, rules, and records while specialized workers handle generation. Add an independent service when it needs separate scaling, ownership, release timing, or isolation. Every network boundary also introduces authentication, deployment, tracing, and partial-failure work, so that boundary should solve a concrete problem.
Architecture and data flow
- Authorized request
The application checks the actor, source scope, input, and request key.
- Job and outbox
A transaction records accepted work and an event to dispatch.
- Bounded worker
The worker calls the provider, validates output, and records progress.
- Persisted result
The interface reads the result and its review or exception state.
The database owns application state. The queue transports work; the provider produces a candidate result. Access checks, budget accounting, and approved actions remain application responsibilities.
Accept a request without losing its dispatch record
Write the job and its outbox event in the same transaction. A separate dispatcher publishes committed events. Scope a request key to the workspace and check that a repeated key still refers to the same input.
function acceptWork(actor, input, requestKey):
authorize(actor, "workflow.submit")
inputHash = hash(normalize(input))
transaction:
existing = findByKey(actor.workspace, requestKey)
if existing:
require(existing.inputHash == inputHash)
return existing.receipt
job = insertJobUnique(
actor.workspace, requestKey, inputHash, input
)
reserveBudget(job.id)
appendOutbox("job.ready", job.id)
return job.receiptEnforce the workspace/key pair with a database uniqueness constraint and retry the transaction on a concurrent key conflict. The dispatcher and consumer must tolerate repeated delivery. Do not hold this database transaction open while calling a model or another remote service.
Event planning with separate plan and image work
Ideavent connects an event brief to a structured plan and supporting imagery. Plan generation and image generation have separate worker paths, while Story and Blueprint present the same plan for different tasks. That separation lets the application represent progress instead of requiring the browser to own the entire generation process.
The architectural lesson is to design the shared record and the operating states before adding more model calls. A plan can be ready while some imagery is still processing. The interface, credit accounting, and recovery behavior all need to agree on what that partial completion means.
Explore the reviewed product architecture and generation workflow in the Ideavent case study. Read the Ideavent.ai case study.
Failure cases to design for
Two writes with no durable handoff
Saving a job and publishing a message as unrelated operations leaves a failure window. Record dispatch intent with the job, then recover unpublished events through a dispatcher.
A timeout treated as proof that nothing happened
A remote task may finish after the caller stops waiting. Reconcile the request’s state before repeating a costly generation or allowing a second external effect.
Progress used as the source of truth
A progress message can be delayed or missed. Reopening the page should reconstruct the job from durable state, with streaming or polling used to update the presentation.
Measurable outcomes
Agree on a baseline, a representative workload, and the evidence needed for the next decision. Keep the measurement conditions with the result.
- Time to a reviewable result
Measure acceptance-to-ready duration at the median and 95th percentile, including queueing, retries, and validation.
Use the observed tail and input-size distribution to choose between an immediate response and a durable job.
- Recovery and repeated effects
Interrupt a worker and repeat delivery; count recovered jobs, duplicate provider work, and duplicate committed results.
Require a clear recovery state and investigate every repeated committed effect before expanding the workload.
- Operating cost per accepted result
Divide provider and processing cost, including failed attempts, by results accepted under the agreed review criteria.
Compare the cost with the useful output and human review effort, rather than only counting successful API calls.
For delivery milestones and acceptance criteria, explore Custom software & platforms and AI software engineering, automation & integrations.
Sources and next reading
- Microsoft: queue-based load leveling. How a queue buffers demand and separates arrival rate from processing rate.
- AWS: transactional outbox. The database/message handoff and the need for consumers to tolerate duplicate delivery.
Return to the AI software engineering guide to connect these decisions to the wider product lifecycle.