Case Study
RenovationRoute AI Intel Service
What eight months of production AI taught us
A production AI service running in a live revenue platform since October 2025. The patterns we sell to PE portfolios are the patterns we shipped, broke, and fixed here.
The Problem the Service Solves
RenovationRoute is a marketplace where homeowners post renovation projects and contractors bid on them. The original flow had homeowners filling out a long structured form: rooms, dimensions, finishes, scope. Most homeowners abandoned the form. The ones who finished it filled it out badly. Contractors got bad inputs and produced bad bids. Disputes followed.
The product question was simple. Could a homeowner describe a project in plain language and get a structured scope a contractor could actually bid on. The technical question was whether an LLM could produce that structure reliably enough to put real money on it.
The answer turned out to be yes, but only with the right architecture, the right evals, and a serious set of defenses around the model. That work became the AI Intel Service.
Architecture
A standalone microservice. Carved out from the Rails monolith on purpose.
Python and FastAPI
The rest of RenovationRoute is Rails. The AI service is Python because the LLM tooling ecosystem lives there and Python is the right fit for an inference-heavy workload that needs different scaling rules than the monolith.
Async by default
User-facing flows do not block on the model. The Rails monolith enqueues work through Sidekiq and Redis, and the Python service does the inference out of band. The user gets a working UI while the AI does its work. Outputs stream in or arrive as notifications.
Its own deployment
The AI service deploys independently from the monolith. Different scaling rules, different failure modes, different release cadence. When the model vendor has an outage, the rest of the platform keeps running.
Clean integration boundary
The monolith calls the AI service through a small typed interface. The service can swap models, swap vendors, or restructure its internals without touching the monolith.
Decisions That Mattered in Production
Evals before prompts
The first version of the AI Intel Service got the eval framework before it got serious prompt work. Without a way to measure whether a change makes the output better or worse, every prompt change is a guess. With evals, prompt iteration becomes engineering. This is the pattern we bring to every sprint.
Latency, quality, and cost are one tradeoff, not three
Bigger models produced better scope output but were too slow for the user flow and cost too much per request. Smaller models were fast and cheap but skipped details. The shipped system uses a tiered approach: a fast cheap model handles classification and routing, a more capable model handles the structured output. Every new use case starts from this question, not from "which model should we use."
Output validation is not optional
The model is wrong sometimes. Structured outputs validate against a schema before they reach the rest of the platform. Failed validations route to a retry with a corrected prompt, then to a human review queue if retry fails. Without this layer, bad outputs would have leaked into bids and into payment flows.
Prompt injection is a real category, not a hypothetical
Homeowners describe projects in free text. That text becomes part of a prompt. A determined user can write input that tries to flip the model into doing something other than scoping. The shipped system separates untrusted user text from instructions, validates the structured output regardless of what the input said, and refuses to take actions that were not part of the original API contract. This is one of the things our AI Diligence engagement looks for in targets, because most teams have not done it.
Cost controls run live, not at month end
Per-request, per-user, and per-day spend caps are enforced in the gateway, not after the bill arrives. Alerts fire on unusual traffic before the cost shows up. The cost story for an LLM service is "no surprises" or it is a problem.
Fallbacks for when the model is wrong, slow, or down
The model fails in three ways: wrong output, slow output, and no output. Each one has a fallback path. Wrong output gets caught by validation and retried or queued. Slow output triggers a "we are still working" UI state so users do not bail. No output falls back to the structured form. The AI feature is value-add, not a hard dependency.
What This Teaches Us About Sprints
The AI Intel Service changed the homeowner experience from "fill out a form" to "describe your project." That is product-changing. A summarization feature that nobody opens is decoration.
Picking a model is fifteen minutes. Building the evals, the validation, the fallbacks, the cost controls, the prompt injection defenses, and the observability is the rest of the sprint. Teams that focus on the model and skip the system end up with a demo, not a product.
Prompt injection defenses, output validation, cost controls, fallback paths. These are not a separate workstream you tack on at the end. They are how the feature works.
Some parts of RenovationRoute use no AI at all because the rules-based version is faster, cheaper, and more correct. That is part of the work too. We will tell a portco when AI is the wrong tool for a use case.
The broader RenovationRoute architecture story (payments, escrow, multi-tenant data, layered security) is covered in a sibling case study.
Want this kind of work in your portfolio?
If you have a portco that needs production AI shipped in twelve weeks, or a target you want assessed for AI risk pre-close, that is what we do.