# The Feedback Loop

A capability bundles orchestrations that share an input and output, serves them at scale and benchmarks them live.

- Page: https://perceivo.ai/university/cap-01-the-feedback-loop
- JSON: https://perceivo.ai/registry/film/cap-01-the-feedback-loop.json
- Kind: film
- Revision: 3 (published 2026-10-19, produced 2026-10-18)
- Previous revision 2 (published 2026-10-10): https://perceivo.ai/registry/film/cap-01-the-feedback-loop/r2.json
- SDK version: none

## Validation

- SDK: not applicable (Concept film; no component or orchestration is offered for download)
- Platform: not applicable (Concept film; no component or orchestration is offered for download)

## Transcript

[0:00] Introduction to Capabilities, Episode 1: The Feedback Loop
[0:06] Welcome to Introduction to Capabilities.
[0:09] This series is about the idea at Perceivo's core.
[0:13] Systems that improve by learning from their own experience.
[0:18] Perceivo capabilities are collections of orchestrations that serve a common customer.
[0:24] They share a common input and output, but can have completely different internal structures.
[0:30] Capabilities provide all the tools you need to serve orchestrations at scale, benchmark their results, and use operational performance to improve outcomes.
[0:40] And in this series, we're going to show you how that's done.
[0:44] This is our capability: one endpoint, with its members, its traffic and its health in one place.
[0:51] Our example is Northwind Traders.
[0:54] The company's HR department wants a chatbot that answers questions for employees about the company's policies and procedures.
[1:02] So, we developed two competing architectures.
[1:06] The first one is Plain.
[1:07] It passes the employee's question to a public model, and returns the response.
[1:13] The second one is Augmented.
[1:15] It attaches information from the company policy manual to the user's prompt, before sending it to the public model.
[1:22] To compare the live performance of these two orchestrations, we bundle them into a capability, which lets us load them together behind a load balancer.
[1:31] We load one as the champion, and the other as a canary, and split the inbound traffic fifty-fifty, on a randomized basis.
[1:40] Here's the distribution as the traffic arrives: both architectures, answering the same kinds of questions, side by side.
[1:48] Perceivo tracks every execution: the input, the output, and every step taken in between.
[1:56] These are the capability's logs.
[1:58] Every call to a model is recorded, and every entry carries the run's trace, so you can follow one execution from one step to the next.
[2:07] Let's open one.
[2:08] This is a single call to the public model: exactly what was sent, and exactly what came back.
[2:15] Snapshots record the data itself, as it moves through the orchestration.
[2:20] The first snapshot is the question, exactly as the employee asked it.
[2:25] The last is the answer they received.
[2:28] And every step in between is here too.
[2:31] You don't build any of this.
[2:33] Every run, on every member, is recorded automatically.
[2:38] When each execution completes, a review is routed to a human to score the result.
[2:44] This is the reviewer's queue: one review for every run.
[2:49] Let's complete one.
[2:51] The reviewer sees the question, and the answer the chatbot gave.
[2:55] They're asked to rate the output across five dimensions, on a Likert scale.
[3:00] And submit.
[3:02] That review is now one more signal about this run, alongside everything else Perceivo knows about it.
[3:09] So how do those signals become a single score? You decide.
[3:13] You create an evaluator component with the Perceivo SDK, that expresses your definition of quality, as it relates to this system's performance.
[3:23] Here, we use a simple weighted average of three metrics: the human review, groundedness, which checks that the answer is supported by the policy manual, and latency.
[3:34] Together, they form our own objective function.
[3:38] This formula decides whether one orchestration is better than another.
[3:43] The system may have dozens of ways to accomplish a task, but now it has the means to determine which one is best.
[3:52] This is not arbitrary.
[3:54] It's a critical design decision, because it focuses the entire system on maximizing this weighted average.
[4:01] Everything that follows, every review, every score, every model we train, every decision the system makes, serves the objective you defined here.
[4:11] And once you write your script, you create a component, an evaluation extension that applies this logic to your records.
[4:18] Capabilities let you create custom scoring functions like this one, and use them to score your executions automatically.
[4:26] Your scoring function could be based entirely on human feedback, or on something much more sophisticated.
[4:33] But the evaluation step makes the human the final arbiter of quality and intent.
[4:39] Those scored runs are combined into an execution graph: a labeled data set that ranks operational outcomes.
[4:47] Let's open one.
[4:48] Every record is a real run: what was asked, what was answered, which architecture answered it, and how well it scored.
[4:56] And because Perceivo generates structured, traceable logs, these execution graphs are easy to vectorize, and use to train a model.
[5:06] But train a model to do what?
[5:08] To predict the next most appropriate action, based on your system's historical performance.
[5:14] Those actions are expressed as a policy: a policy that drives the transitions in a finite state machine.
[5:22] In this series, we use our execution history to improve our system in a meaningful way.
[5:28] Instead of splitting traffic fifty-fifty at random, we train a classifier on the scored execution graphs.
[5:35] Its job is simple: decide whether a given question is best served by the Plain orchestration, or by the Augmented one.
[5:43] This is the Intelligence dashboard, where that work happens.
[5:47] It takes three steps.
[5:49] One: load the features into the feature store.
[5:52] Here are the features the classifier learns from, taken straight from the execution graph, and the label: which architecture scored best.
[6:01] Two: train a model to predict which orchestration will produce the better outcome.
[6:07] This is the trained model, and how well it predicts on runs it has never seen.
[6:12] Three: serve it behind a secure endpoint, so we can start getting those predictions.
[6:19] Then we take out the load balancer, and put a finite state machine in its place.
[6:24] Its policy has one tool, and that tool calls the trained classifier.
[6:29] The policy takes the classifier's answer, and tells the state machine which state to move to: Augmented, or Plain.
[6:37] That decision isn't random.
[6:40] It's the statistically sound choice, based on the available feedback.
[6:45] And here it is in Perceivo, as an orchestration of its own.
[6:49] We think this is better. But we're not sure.
[6:53] So rather than replacing the load balancer entirely, we put our hypothetically better design behind the load balancer as a canary, and send it ten percent of the traffic to experiment with.
[7:04] If it truly is better, we'll see it in the next round of evaluations.
[7:09] Here it is live.
[7:10] Questions arrive at a steady pace, shared across all three, and the queue keeps up.
[7:16] And we can confirm the result through the same benchmarking process we used to train the classifier.
[7:22] This is a relatively simple A/B decision.
[7:26] We'd expect questions about the policies to be best answered by referencing the actual manual, which the Augmented orchestration does.
[7:34] And we'd expect questions that don't relate to the policy manual to be best answered without adding policy context to the prompt.
[7:42] But the concept is profound.
[7:45] Learned functions can easily be incorporated into your orchestrations, as advisors or decision makers in the reasoning process.
[7:53] These functions aren't trained to predict the next word in a sentence.
[7:58] They're trained on your operational data, to choose the sequence of actions that maximizes performance.
[8:05] And scored execution graphs can inform much more complex state machines.
[8:10] As we saw in Introduction to Orchestrations, a lot of familiar designs are just different state machines.
[8:17] A conditional router: an if-then policy that sends the work down one path or another.
[8:23] A generator and discriminator: one component produces, another judges, and the loop runs until the work passes.
[8:32] A mixture of experts: several specialists, and a policy that decides whose answer to trust.
[8:39] And hierarchical reasoning: state machines inside state machines.
[8:45] And there are many more.
[8:46] A cascade, that tries a small model first and escalates when it has to. A debate.
[8:53] A planner and an executor. An ensemble vote.
[8:57] A human in the loop.
[8:59] These are the ones we know.
[9:00] The interesting ones are the ones nobody has drawn yet.
[9:05] And AI engineers have an equally large choice of models to base their policy decisions on.
[9:11] We're using a simple classifier in this demo, but scored execution graphs can train random forests, neural networks, or Markov processes.
[9:22] Put it all together, and the capability and the intelligence service become a single cycle.
[9:28] One living system, acting, observing, scoring and learning, all focused on the objective function you defined in the evaluation service.
[9:37] This series has been carefully designed to illustrate the power of Perceivo's integrated platform.
[9:44] It will demonstrate how systems built with it can use operational intelligence to learn and improve from their own experience.
[9:52] We appreciate you joining us on this journey.
[9:55] Everything in this demo is in the Perceivo SDK repository, under examples.
[10:02] Next time: Anatomy of a Capability.
[10:05] We open up a capability, and look at every setting that controls how it runs, scales and learns. See you there.