The companion article shows what the pipeline does for clients. This one is how it is built, the decisions that made it production-grade instead of a demo, and what months of daily use changed about the design. Some of those changes were reversals.
ArchUpgrade takes working 3D renders and produces photorealistic, quality-scored image sets. Daily production use since early 2026. As of September: about 200,000 lines of code, 91 workflow graphs, roughly 750 backend and 560 frontend tests. Everything runs locally: models, GPUs, the quality reviewer. No client image leaves the network.
- Backend: Python 3.11 / FastAPI with an async SQLite database. One process orchestrates job queues, render dispatch, file management, delivery packaging, and the WebSocket stream the interfaces watch.
- Frontends: three Vite/React apps. The operator UI with its numbered workflow (Ingest → Configure → Queue → Review → Upscale → Deliver, plus tool pages), a client order portal, an admin portal.
- Rendering: SwarmUI instances driving ComfyUI on the GPU nodes. Image-to-image enhancement is built on the Qwen image-edit family with my own photorealism LoRA; sibling graphs cover text-to-image, multi-reference editing, seven upscalers, five lighting tools, camera-angle passes, inpainting and outpainting, mesh and Gaussian-splat generation, video, speech and music.
- Quality control: a local vision LLM via Ollama scores every generated image. No API keys, no per-image cloud cost.
- Hardware: a small farm of my own GPU nodes, 16 GB cards for enhancement batches, one 48 GB card for heavy upscales and video. The farm doubles as the render farm at night; a separate tool switches the nodes between the two jobs.
The central screen is Versioning: source images left, iteration parameters middle, a live preview of the exact structured prompt on the right.
Image-edit models respond much better to terse, keyed instructions than to walls of prose. I learned that by writing the walls of prose first. The assembler merges per-image notes, per-directory context, reusable preset cards and parameter categories like season or budget tier into one structured payload with deterministic ordering. Same inputs, same prompt, every time. The rendered prompt is stored on every version as the exact string sent, never reconstructed from templates afterwards.
Parameters are organised into iteration axes. Two lighting moods, three furnishing styles, five seeds: the planner expands the cartesian product into thirty versions, each labelled with exactly how it was made.
Every output gets a label like g03_i_s02_e01: generation 3, iteration code, seed 2, edit 1. Retries, mutations, manual edits, upscales, crops, relights and camera moves each append their own letter, so any image’s ancestry reads from its filename. Re-queueing bumps the generation number. Nothing on disk is ever overwritten.
The review screen renders this as a tree: source render at the root, generations as children, every derived operation branching below.
This sounds like bookkeeping until a client says “we liked the third version from last month, can we see it in evening light?” and you branch from that node instead of reconstructing settings from memory.
Every completed render is evaluated by a local vision LLM against the source image and the prompt. Five independent 0–10 scores (aesthetics, fidelity, prompt adherence, technical quality, style match), written notes, a list of missing elements. An image passes only if every aspect clears its threshold; failures are retried with correction notes fed back into the prompt.
The scores are advisory. A human still picks. But they invert the economics of review: instead of inspecting 300 images I inspect the 40 that passed, sorted by score. On a recent 18-view interior project the system generated over 300 versions, the LLM filtered, I picked 18.
Two rules learned the expensive way, now in every judge in the system: ask about presence, not absence (“is there a visible defect?” is answered far more reliably than “is this good”), and turn the model’s thinking mode off for fixed-format answers, which took a verdict from twenty-five seconds to under one.
Enhancement is one graph among many. Once a version is picked, targeted passes run on it without a trip back to the 3D scene: five lighting tools (scene-locked time-of-day relight, light migration from a reference photo, multi-angle key light, studio delight, light restoration), sky replacement and material swap, generative expand, upscaling, and a camera-angle pass for a slightly different viewpoint. Each is its own letter in the tree and its own row in the queue.
That relight came off the farm through the same API my automation uses, from an eight-year-old archive render, in about half a minute once the model was loaded. The prompt the tool assembles opens with a scene-lock clause that pins the viewpoint before it says a word about light. That clause, learned from a dozen drifting outputs, is most of the engineering.
Dispatch runs across the SwarmUI/ComfyUI backends with weighted round-robin. Each machine declares roles (image editing, text-to-image, upscaling, 3D, video) with per-role weights, so the 48 GB card takes heavy upscales while smaller cards chew through enhancement batches. Health checks gate offline machines out of rotation; a crash-recovery sweep on startup means a power blip never leaves the queue corrupted. Eight runners share one rule: acquire the GPU semaphore before popping the queue, so the “pending” count is honest.
Every model, LoRA, checkpoint and external tool carries a licence class (permissive, copyleft, non-commercial, unlicensed), declared once in configuration. Every produced asset records the worst-case class of any tool that touched it, plus the tool list, on the database row and in a JSON sidecar next to the file. A single deployment switch, allow_noncommercial, hides every non-permissive backend from the interface and refuses it at the API. Flipped off, the system physically cannot ship an image made with a research-only model. Several of the most impressive open models are non-commercial; the vetting database exists to catch that before a model reaches this pipeline.
Months of daily use added the machinery that turns a tool into infrastructure:
- Observability. One immutable row per terminal render attempt, from every runner’s
finally:block; VRAM and queue-depth samples from every node every five minutes; per-node roll-ups and a “retry everything that failed for a safe reason” button. - Retention and backups. Sandbox output older than a retention window is reconciled and cleared through dry-run → checked list → apply, with “the delivery tree is unreachable” written in code, not configuration. The database is backed up with SQLite’s online backup API and integrity-checked, never file-copied.
- A disk guard at every runner’s enqueue that refuses new renders below a free-space floor rather than corrupting a batch halfway.
- Auth and a client loop. Principals and roles (owner, studio, service, client), bearer tokens for automation, per-order links for clients, watermarked previews, one open revision per order, pin comments normalised to the image, a client zip that carries no studio paths. A LAN peer with no credential is still the owner, so the trust model did not change; every existing test passed untouched when auth landed.
- A render facade. One path-in, path-out API through which every other process in the studio renders (scheduled agents, scripts, other repositories), with no database rows and no UI state. That boundary is why the agents could move into their own project without importing a line of this one.
- Two machines. The dev box runs
main; the production server runs astablebranch off a shared output store. Promotion is a deploy script, not agit pullon the box clients depend on.
The final step packages picked versions with clean sequential filenames while a metadata file preserves the internal labels, generation parameters and provenance. Per-category quotas (“15 exteriors, 3 interiors”) are enforced before the package can ship.
- Determinism is the feature. Seeded generation, snapshot-at-queue-time configuration, labels that encode ancestry. That is what makes it a production tool rather than a slot machine.
- Score with a local model. Quality review is the highest-volume LLM call in the system. Local makes per-image cost zero, so you never hesitate to review everything.
- Plan for failure everywhere. GPUs go offline, processes restart mid-batch, JSON files corrupt at the worst moment. Atomic writes, per-cell failure isolation, mark-failed-never-auto-retry. Unglamorous and essential.
- Licences are a schema field. If the class is not on the row, it is not in the delivery, and someone finds out after the brochure is printed.
- Verify on real hardware. Mock mode proves the UI-to-queue-to-database circuit and nothing about the render. Six shipped features had no interface until an audit found them, and a generative-expand tool was dark for two months because it sent a mask to the wrong graph. Both found by running the real path and looking at the picture.
- The model is maybe a fifth of the system. The rest is version management, prompt assembly, queueing, review UX, delivery, licences, observability. The machinery that turns a great demo into a tool a studio uses every day.
ArchUpgrade is internal tooling, not a product, though its finishing pass now runs on other studios’ renders through ai.visiongraphics.eu. The pipeline auditing and automation work I do for other studios grew out of building it. If you are wiring AI into a real production pipeline and want to compare notes, get in touch.