Back to Blog
AI Agent Observability: What to Track and How to Start

AI Agent Observability: What to Track and How to Start

AI Agent Observability: What to Track and How to Start

Engineer reviewing an AI agent execution trace

AI agent observability means recording every step an agent takes in production (model calls, tool calls, handoffs, outcomes) and judging whether each step was right, so your team can explain and fix its behavior. Traditional monitoring tells you a service is up and fast. It does not tell you whether the agent’s plan or answer was correct. This guide covers what to capture, which metrics matter, how agents fail, how to test them, and what French and EU rules expect from your logs.

Botiqueai
Build More Observable AI Agents
BotiqueAI creates custom intelligent agents and AI automations tailored to business needs, helping teams improve operational efficiency and customer relations.
Explore BotiqueAI

TL;DR

  • Agent observability combines telemetry (what happened) with evaluations (was it right). Latency and error rates alone are not enough.
  • Trace every run as a hierarchy: the agent invocation, each model call, each tool call. Measure at session, trace and span level.
  • Track task success, tool-call accuracy, cost per trace, loops and groundedness.
  • Test with injected faults before launch, then keep evaluating in production, because agent quality drifts over time.
  • In France, keep logs useful but proportionate: justified retention, traceable decisions and human approval for higher-risk actions.

Table of Contents

What AI agent observability actually is

An agent is not a single model call. It plans, calls tools, reads results, revises and sometimes hands work to another agent. Observability for it means capturing that whole chain and connecting it to the outcome the user cared about.

Two ingredients make it work. The first is a structured record of each run. The second is judgment attached to that record. According to Splunk’s practitioner guide, the judgment comes from evals, an evaluator plus the score it produces, attached to the same records. Together they answer the question a stack trace cannot: why did the agent choose that?

How it differs from LLM and software observability

LLM observability watches individual model calls: prompt in, response out, tokens, latency. Agent observability follows calls, tools and handoffs toward a goal. A model call can look perfect while the agent still picks the wrong tool two steps later.

LLM call monitoring compared with agent observability

The deeper difference is determinism. Microsoft’s guidance on observability for generative and agentic AI notes that conventional software has largely predictable execution paths, while AI systems are probabilistic by design: outputs vary from run to run. Observability therefore has to extend beyond latency, errors and throughput to include evaluation and governance.

IBM Research’s Dany Moshkovich makes a related point in the AgentOps paper: observing an agentic system goes beyond classical tracing. You must capture how decisions and execution flows emerge dynamically, including code the agent generates at runtime, and the feedback loops (reflection, guardrails, user input) that shape behavior.

Why you need it once agents reach production

A demo agent is watched by the person who built it. A production agent talks to customers, touches your CRM and spends money while nobody is looking. Several people then need answers from the same system.

The AgentOps framework identifies four roles engaging with agents at different points of the lifecycle: developers, testers, site reliability engineers and business users. It proposes a six-stage pipeline: behavior observation, metric collection, issue detection, root cause analysis, optimized recommendations and runtime automation.

In practice that means:

  • Developers need to see why a run took a wrong turn.
  • Support and operations leads need to know how many conversations the agent resolved and how many it escalated.
  • Finance needs cost per task, not just a monthly model bill.
  • Compliance needs to reconstruct what the agent did with personal data.

If your agent is a customer-facing chatbot, the stakes are visible fast. Our article on preventing AI hallucinations in your business covers the answer-quality side of the same problem.

What to capture: traces, spans and context

Start with the trace. Microsoft recommends evolving logs, metrics and traces to capture request identity context, timestamps, conversation and run identifiers, user inputs, system responses, retrieval source provenance, and every agent or tool invocation, including tool names, arguments, permissions and outputs. It also recommends representing traces and metrics with OpenTelemetry GenAI semantic conventions.

Those conventions define a simple hierarchy, as described in Augment Code’s monitoring guide:

  1. A top-level invoke_agent span for the whole run.
  2. Child chat spans for each LLM call.
  3. execute_tool spans for each tool invocation.

Core attributes include the requested model and the response finish reasons. By default, instrumentation captures no prompt content or tool arguments because of sensitive data concerns. That default is a good starting point in France: switch content capture on deliberately, for defined purposes, not by accident.

Three levels of measurement

The same guide describes measuring at three levels:

Level What you look at Example questions
Session Resolution rate, escalation frequency, completion rate Did the customer get what they came for?
Trace Step efficiency and correctness Did the agent take a sensible path?
Span Each tool call, API request and reasoning step Which call was slow, wrong or repeated?

Session metrics tell the business whether the agent works. Span metrics tell engineers where to fix it. You need both, linked by the same run identifier.

The metrics that matter

Augment Code groups agent metrics into five families. Here they are, with how we would use each one.

  • Task success and goal accuracy. The headline number. Define “success” per use case before you launch: an order status answered correctly, a lead qualified, a ticket routed.
  • Tool-call accuracy and recall. Did the agent call the right tool with the right arguments, and did it call the tools it should have?
  • Cumulative cost per trace. Token spend summed across every model call in a run. Averages hide the expensive outliers, so watch the tail.
  • Loop detection. Repeated-span patterns reveal an agent stuck retrying the same step and burning budget.
  • Groundedness and hallucination rate. Measured through rule-based checks, LLM-as-a-judge scoring and human annotation, ideally combined.

French teams can also draw on the Hub France IA white paper on evaluating agentic AI (2026). Its analysis grid covers concepts such as the agentic loop, tool calls, traces and trajectories, and metrics including result quality, step efficiency, costs, latency, tool choice and resilience. It also covers evaluation methods (trajectory analysis, simulations, production evaluation).

A practical tip: keep the number of dashboard metrics small. Five to eight metrics that someone reviews weekly beat forty that nobody opens. Our two-phase playbook for LLM evaluations shows one way to build the evaluation set behind them.

Common failure modes and how to spot them

You cannot monitor for failures you have not named. Three taxonomies from recent research give you a vocabulary.

Failures at the task, component and feedback level

The AgentOps paper categorizes failures into three groups:

  • Task failures: incomplete tasks, incorrect or partial outputs, and responses that are syntactically valid but unusable.
  • Component issues: LLM timeouts, low-confidence outputs, tool or vector database errors, and agents that miscoordinate.
  • Feedback-handling failures: guardrail violations, rejected reflective feedback, user dissatisfaction and collaboration failures.

Tool-level failures

The ChaosLLM work presented at ISSRE 2025 defines four generic failure modes for tool-calling agents: the tool is unreachable, the response is slow, the tool hangs with no response, or the response is plausible but semantically wrong. The last one is the dangerous one, because nothing throws an error.

The AgentCheck workbench shows how real that is. Across five agents it re-ran scenarios with perturbed tool responses. The failures were usually silent: the agent confidently used incorrect tool outputs rather than crashing. Your observability should therefore compare what a tool returned with what the agent said afterward, not only count errors.

Why final-answer checks are not enough

A 2026 benchmark paper on fault localization, AgentChaosBench, argues that reliability belongs to the whole execution (tool calls, model calls, guardrails, inter-agent messages), not to the final answer alone. It also reports that automated diagnosis from a single trace is still hard: zero-shot models reached only modest accuracy at naming the fault type. Treat automated root-cause suggestions as leads for a human, not verdicts. For a broader view of why this matters for autonomy, see why reliability is the new autonomy.

Testing reliability before launch, and watching for regressions after

Observability is not only a production tool. The same traces feed your pre-release tests.

Inject faults before users do

Fault injection is the most transferable idea in the research. AgentCheck runs an agent against its real tools, records every tool response, then replays the run with a response altered by one of 12 fault types. The best of five agents passed 105 of 120 scenarios, the weakest 77, according to the AgentCheck paper. You do not need the workbench to copy the principle: break a tool on purpose in staging and see whether your agent notices.

A trace-based assurance framework goes further. It records executions as message-action traces with step and trace contracts that give machine-checkable verdicts, point to the first violating step and support deterministic replay. It also includes stress testing and fault injection at service, retrieval and memory boundaries.

Test with simulated tools, not only real ones

The AgentInspect framework, presented at ISSTA 2026, detects six types of behavioral failures in LangChain-based agents. Its evaluation found that simulated and hybrid settings expose failure modes that never appear in baseline runs with real tool responses. In other words, a clean run against healthy tools proves little.

Expect agents to age

Even with frozen model weights, an agent’s effective state changes. The AgingBench authors describe four aging mechanisms: compression aging (summaries drop details that matter later), interference aging (similar memories crowd out the right fact), revision aging (changed state not updated correctly) and maintenance aging (routine lifecycle events trigger regressions). Behavioral tests can stay green while factual precision decays.

The practical consequence: schedule recurring evaluations on a fixed golden set, and compare scores over weeks, not only at release. If your agent keeps memory across conversations, add long-session scenarios. Multi-turn degradation is a known trap, which we discuss in why multi-turn performance matters.

Logging, privacy and human oversight in France

More logging is not automatically better. Prompts and tool arguments often contain personal data, which makes your observability store a GDPR-relevant system in its own right.

The CNIL’s guidance on using an AI system in production says that logging to support explicability should not come at the expense of the privacy of operators or data subjects. It asks whether logged elements are kept for a justified period and limited to the categories needed to explain a decision.

For security logs, the CNIL’s GDPR security practice guide recommends:

  • keeping logs for a rolling period between six months and one year;
  • recording the creation, consultation, sharing, modification and deletion of data, with the author’s identifier, date, time and nature of the operation;
  • analyzing logs in real time or at short intervals to detect incidents.

The CNIL also describes three levels of human oversight: human-in-the-loop, human-on-the-loop and human-in-command. Pick the level per action, not per agent.

On agentic AI specifically, the CNIL and the French AI and Digital Council (CIANum) issued a joint exploratory note in July 2026. According to a summary of the note, it identifies the proliferation of memory instances as the main source of opacity. It recommends traceability that lets you reconstruct a whole decision workflow, granular user controls over data access, risk-tiered actions with human approval for higher-risk ones, and a user-accessible kill switch. Those recommendations map neatly onto good observability design: trace everything needed to replay a decision, and gate the risky steps.

Practical choices that follow:

  • Capture identifiers and metadata by default, and prompt content only where you need it.
  • Mask or pseudonymize personal data before it reaches the trace store.
  • Set a retention period per log type and delete on schedule.
  • Restrict who can read traces, and log that access too.

For your specific data flows, involve your data protection officer.

Our perspective

We build chatbots, agents and automations for businesses, and the pattern we see is consistent: teams invest heavily in getting an agent to work once, then discover they cannot tell whether it still works in week six.

That is why we treat observability as part of the build, not an add-on. Every agent we design is meant to leave a readable trail: what the user asked, which tools were called, what came back and what the agent finally answered. Without that trail, a wrong answer is an anecdote. With it, a wrong answer is a ticket someone can fix.

Our approach starts small. We define what “success” means for the use case with the business owner, build a proof of concept, and instrument it from the first day so that the evaluation set grows from real conversations. For customer-facing agents we pair this with clear escalation rules to a human, because the cheapest failure is the one the agent hands over instead of improvising.

We also design for GDPR from the start: minimal logging, justified retention and controlled access to traces. A reliable agent and a compliant agent are built the same way, with traceability and human oversight where the stakes are high.

Our view in one line: an agent you cannot observe is an agent you cannot responsibly give more autonomy.

Botiqueai

Put observability into your next agent project

If you are planning a chatbot, a workflow agent or an automation connected to your CRM or ERP, we can build it with tracing and evaluation in place from the proof of concept. We state that we can often deliver a proof of concept within 14 days, offer a free initial audit and require no long-term contract. Our custom chatbot, AI integration and automation services are the place to start.

Botiqueai

For a customer-facing assistant on a website or store, see how our team approaches the custom AI automation offer.

A starter plan for the first 30 days

  1. Define success. Write down two or three outcomes per use case and who owns each number.
  2. Instrument the run. Emit one trace per run with spans for each model and tool call, using OpenTelemetry GenAI conventions where your stack supports them.
  3. Build a small golden set. Start with 30 to 50 real or realistic conversations and score them with a mix of rules, an LLM judge and human review.
  4. Break things in staging. Make each tool slow, silent or wrong, and see what the agent does.
  5. Set alerts on the tail. Alert on loops, cost outliers, escalation spikes and drops in task success, not only on errors.
  6. Decide retention and access. Agree log retention, masking and access rules with your data protection officer.
  7. Review weekly. Read ten failed traces together. Add each new failure pattern to the golden set.

If you are still choosing which kind of agent to build, our overview of enterprise AI agent types helps you match the use case to the level of oversight it needs.

FAQ

What is the difference between AI agent monitoring and AI agent observability?

Monitoring watches known signals such as latency, errors and cost against thresholds. Observability keeps enough structured detail (traces, tool arguments, scores) that you can investigate questions you did not anticipate, such as why the agent chose a particular tool. In practice teams need both: monitoring raises the alarm, observability explains it.

Which metrics should I track first?

Start with task success rate, tool-call accuracy, cost per trace, loop detection and a groundedness check. Add session-level numbers such as escalation rate and completion rate so the business can read the results. Keep the set small enough that someone actually reviews it every week.

How do I test an agent’s reliability before deployment?

Run it against a fixed set of realistic scenarios, then repeat with tools deliberately made slow, silent or incorrect. Research shows that simulated and perturbed tool responses reveal failures that clean runs hide. Keep the scenarios and re-run them after every prompt, model or tool change.

Can I log full prompts and tool outputs under GDPR?

Only where it is justified and proportionate. The CNIL asks that logged elements are kept for a justified period and limited to what is needed to explain a decision. Default to metadata and identifiers, mask personal data, and discuss any content capture with your data protection officer.

Do I need human oversight if the agent is well monitored?

For higher-risk actions, yes. The CNIL describes human-in-the-loop, human-on-the-loop and human-in-command models, and the 2026 CNIL and CIANum note recommends human approval for higher-risk actions. Monitoring tells you what happened. Oversight decides what is allowed to happen.

Sources

© 2026 BotiqueAI — Reproduction prohibited without attribution.