From PoC to Production

Between May and June 2025, I wrote a series of posts on LinkedIn about the reality of designing, evaluating, and scaling multi-agent AI systems. This article is a direct regrouping and rephrasing of those posts, with no added or removed content — only organized for clarity.

📌 Table of Contents

1. Evaluation is your main driver
2. The real battle begins in Production
3. Regression Testing and Guardrails: The Hidden Heroes
4. Evaluation inside the Agentic Loop
5. Building for Scale and Automation

1. Evaluation is your main driver

Most tech teams have been missing a key point when evaluating AI systems

It's not new. Even back in the classical ML days, I noticed this pattern:
we stuck to abstract, generic metrics (accuracy, F1-score, etc.) instead of designing domain-specific ones that truly reflect business value.

Today, I'll go even further:
Build client-specific metrics: metrics that align exactly with your client's goals and values.

Spoiler: these often boil down to money — ROI.

Concrete example. Ask business owners which scenario they'd choose:
1️⃣ A sales AI that hallucinates 10% of the time, but closes 20% of deals
2️⃣ A sales AI that hallucinates 1%, but only closes 1%

The answer isn't obvious, right? It's not that easy, since you do not know the real impact in terms of costs and revenues for each scenario...

Can LLM-as-a-judge be considered a form of evaluation?

As a data scientist, my answer is NO. At least, not in the strict sense of the word.

Here's why:
Evaluation without a gold standard is just another opinion.
Or if you prefer: it's more of an analysis than an evaluation.

Don't get me wrong! I'm not saying LLM-as-a-judge is useless. It can provide useful signals to improve your system, especially when human evaluations aren't scalable.

But calling it evaluation is misleading, because you're not comparing to a source of truth.

As a data scientist, I've already seen this kind of confusion, especially when discussing unsupervised learning evaluation.

If you've worked in data science, you know:
Evaluation in unsupervised learning is tricky. You often have to tie it back to a supervised task or a business outcome to assess its value.

Same goes for evaluating text generation.

So how do we handle this properly?
✅ Define expected outputs at the data point level so you can apply real metrics.
✅ Don't ask the LLM to reason, ask it to compare your system's output against a reference.
✅ Let them help you understand why an output deviated, not just whether it's good or bad.
✅ Instead of relying on end-to-end judgment, look at what each agent or tool is doing in your system.

What does it mean to evaluate RAG? Is it worth doing?

How do we break it down into multiple steps, and what's the most critical part?

First, understand that RAG isn't just a text-based system. It can involve various data types and formats integrated into the LLM's context.
RAG can cover API calls, SQL databases, web searches, and of course vector databases.

To control your agent's output quality, you must ensure quality at every process step.

In the agent space, retrieval is critical, because it involves multiple sub-steps:

• The agent builds the query itself. You need to assess whether it's doing this correctly. ex: not misusing database tools or accessing unauthorized data.
• Evaluate the real targets (entities, lines, documents, or chunks) that should be retrieved and used as context. Then compute metrics like false positives and false negatives.
• Problems can also come from your knowledge base. You need clean, curated data, especially during the building and testing phases.
• And ensure you're building efficient context retrieval! If the data you're feeding the LLM is too big (and isn't focused on what needs to be shown to the user) you'll lose your LLM's attention, even with models that support large contexts.

Then, assess the generation part. 3 key points to evaluate:

• Hallucination and Non-Based Information: Is the output strictly based on your RAG context? Errors here could come from model quality, prompt design, or lack of guidance.
• Relevancy: Does the generation highlight the right parts of the context? Is it structured as expected, per prompt policy or fine-tuning?
• Exhaustivity: Is the generated text exhaustive? Be cautious, some filters in the prompt might deliberately exclude parts of the retrieved data.

RAG is the most important part to evaluate in your system. It's where most issues originate. Break it down and evaluate each part carefully.
✅ Study your errors and keep track of them in neat and well-defined automated tests.
✅ Build a robust test framework with a diverse query set and reference answers.
✅ Use metrics like precision, recall, and context relevance for retrieval.
✅ For generation, focus on faithfulness and answer relevance.
✅ Regularly update your evaluation datasets based on real-world usage.
✅ Use LLM-based evaluators to scale your assessment efficiently.

---

2. The real battle begins in Production

Your AI system just hit production. Now comes a new challenge: not breaking it

Congratulations! 🎊

You've shipped your wonderful AI platform to prod, and it's working pretty well for a first version.

And now comes the hard part: you need to add all the features that weren't in your MVP, the ones that are crucial to keeping your AI services ahead of the competition.

And guess what? It's not going to be a small iteration.

These new features will likely require a structural review of your agent orchestration, and even a rethink of your entire multi-agent system architecture (Yes, there are multiple architectures with new ones introduced every week.)

We can say it:
Your production is legacy in less than 3 months. 😅

This isn't entirely new…
But two things are probably different from your usual tech stack:
1- Legacy happens fast in AI.
Some projects become legacy before even hitting production.
We've experienced this ourselves. When a framework we were using got a major upgrade, we had no choice but to migrate if we wanted to keep building new features.
2- Migration is riskier.
You have less control over your AI platform than with a deterministic system.
Which means even small changes can lead to unexpected behaviors.

And here comes regression testing.
Regression tests are your best ally to make sure refactors or architecture changes don't break critical production features.
✅ Create and maintain dedicated regression scenarios.
✅ Define metrics that qualify outputs: since you're working with AI, pass/fail isn't enough.
✅ Automate these tests or every release will turn into days of manual QA and retesting.

If you're a startup moving fast in AI with users already relying on your first product:

Invest in regression testing now!

Otherwise, your roadmap will slow down before it even picks up speed.

Improving your AI system isn't just about development time.

It's far more correlated with the number of real cases processed in production.

Why?

Because evaluation isn't meaningful until your system has faced a wide range of inputs... and even then, some edge cases won't appear until you're operating at scale.

Here's what we've observed again and again:

✅ Complex agent tasks can perform flawlessly in dev, but start breaking down under production load or when inputs deviate even slightly from the original distribution.
✅ Free text input is unpredictable. No matter how creative your dev team is, users will always surprise you.
✅ It's not just LLMs: rate limits, data quality issues, and fragile pipelines will all show up once the volume kicks in.
✅ Even your data pipelines need to be stress tested at scale. You'll spot performance cliffs and resilience gaps only under pressure.

Scaling to production gives your team real signals:

• What breaks?
• Where is the system fragile?
• What improvements would make the most users happy?

With enough volume, your team gets the feedback it needs to iterate faster and with higher confidence.

Scaling isn't a nice-to-have.
It's the prerequisite for building reliable and evolving AI systems.

Production support and online evaluation are not just ops tasks, they're strategic levers for your team.

After months running AI systems 24/7, one thing is clear:
Production is where you build real intuition for your next version.

Evaluation before prod is critical, but watching real user interactions is what sharpens your instincts and helps prioritize what matters.

Here's how we turn production monitoring into a superpower:

🔁 Make it a team-wide habit: Everyone in the team should get a taste of real production data. It fuels both creativity and clarity.

📈 Use it to tune priorities: Support helps reveal what truly matters for the business and what's just "nice to have."
When building fast, you want to deliver what's essential, and hold off on the rest, especially since priorities often shift between project and production phases.

💡 Help your team develop good trade-offs: Not every issue needs an immediate fix.
But sometimes a quick patch can avoid user frustration, even if it's not perfect.

📊 Quantify issues: Log incident frequency and business impact. This data will sharpen priorities and guide improvements, especially for your guardrail, the most critical component of your system.

---

3. Regression Testing and Guardrails: The Hidden Heroes

Your guardrail is the most critical part of your multi-agent system. It's your system's gateway. Your first and last line of defense.

First line (keeps the chaos out) when it's the input guardrail:

• helps you catch out of scope/unexpected inputs (things your workflows weren't designed to handle). With these kinds of input, you're almost guaranteed to get unreliable, unpredictable results.
• It's also your security guard: detecting jailbreaks, prompt injections, and malicious inputs.

Last line (keeps the chaos in) when it's the output guardrail:

• It helps drastically reduce hallucinations.
• It can flag out-of-scope topics that slipped past the input guardrail.
• It should also enforce brand moderation, keeping responses aligned with company image and policies.

Think of building a reliable guardrail as a hard classification problem. Like any classification task, you're tuning the threshold between false positives (blocking good inputs) and false negatives (letting bad ones through).

The challenge?
Your input and output is human language: open-ended, messy, context-dependent. The distribution is huge, and the ambiguity is real. There's no perfect line.

Here are some tips if you're tackling this problem:
✅ Define clear and concise policies: Set strong guidelines for how your guardrail should behave.
✅ Inject necessary information: Feed the guardrail only with data needed to make the right decision.
✅ Define good and bad examples: Help it learn what should be let through - and what shouldn't.
✅ Test different LLM models: Your guardrail is your most important agent. Pay the necessary price to have best quality.
✅ Evaluate performance: Use metrics like accuracy, recall, and precision. Better yet, build custom metrics tied to business impact.

Solid guardrails are what separate valuable AI systems from chaotic ones.

---

4. Evaluation inside the Agentic Loop

Data extraction evaluation matters in agentic systems

When building an agentic system, it's important to have data points that can be checked at the heart of your process to ensure you're on the right path.

Think of these as intermediary milestones, deliverables you're asking your multi-agent system to produce on its way to the ultimate objective.

These checkpoints serve a dual purpose:

• Tangible Measurements: They provide clear metrics that simplify system monitoring and facilitate the evaluation of each agent's performance.
• Process Insights: They offer valuable data that can enrich your understanding of the automated process and even serve as a source of structured knowledge about your users.

Moreover, these data points should be used to establish hard controls, dictating actions that agents must or must not take based on the extracted information.

Additionally, this metadata functions as a form of memory. Ensuring that agents get this information at critical steps of the process and that it's never lost.

Want more reliable agents?
Start designing with these internal checkpoints in mind.

---

5. Building for Scale and Automation

R&D is not optional. Be ready to pay for experimentation.

If you want to bring cutting-edge AI systems to production, you'll need to build an R&D mindset into your team and invest in experimentation.

The challenge?
Keep experimentation fast and affordable.
You want feedback in days, not weeks.

And just like any expert team, to move fast you need the right tooling.
As your production matures and your agent tasks become more complex, your tools need to scale with you.

At NORMA, when we started building multi-agent systems, we hit a clear bottleneck: Maintaining constant quality was slowing down our shipping velocity.
So, we built something.

A tool to test AI agents end to end, at scale.
It started as an internal utility.
Today, it's a full platform for teams who need to control agent quality over time and evaluate changes across versions efficiently.

You want reliable and efficient AI?

Make testing easy for your team!

Why testing is harder in AI platforms:

• Your input is challenging to control and fully cover.
• LLMs are everywhere in your pipeline, each bringing potential variance to your process.
• Many steps rely on structured outputs from LLMs, which aren't always easy to obtain systematically.
• Agents may access external sources you don't fully control.
• Outputs vary widely: generated text, finite decision spaces, data extraction...
• AI frameworks themselves are still evolving quickly and might introduce instabilities or deprecate parts of your code overnight.

For all these reasons, developers often struggle to build new features while maintaining high production quality and stability.

Don't over-plan, start automating tests now to relieve your dev team.

Anything that reduces developer workload is worth pursuing. For example:

• Begin with a simple script that automatically runs tests and saves results to an accessible folder.
• Execute these tests in parallel to accelerate feedback.
• Provide a clear dashboard or interface for easily inspecting results.
• Encourage your team to establish metrics to continuously monitor overall quality.

Start by measuring outputs globally, then prioritize evaluating the most critical sub-steps, usually those handling data manipulation.
After several iterations, your developers will become comfortable with evaluation processes, naturally integrating these tests to ensure high-quality features. At NORMA, we’ve even automated evaluation directly into our CI/CD pipeline for every PR 🔥

Check our quick demo here: