Scope & boundaries
This guide covers tracing, evaluation and monitoring for generative systems, where there is no accuracy metric and correctness has to be defined before it can be measured.
It does not cover building and running agents (AI Agent & Agentic AI Platforms), classical model operations and drift (MLOps Platforms), policy, model inventory and regulatory evidence (AI Governance & Responsible AI), or blocking hostile input and unsafe agent actions (AI Runtime Security & LLM Guardrails).
Executive Summary
A trace shows what an LLM application did. An evaluation says whether it did it well. Buy a platform that does both on the same data, under a license and hosting model your security team accepts.
LLM observability is the practice of recording what an application built on a large language model (LLM) does on every request, and measuring whether the result was correct, safe and affordable. A platform for it captures each model call, retrieval, tool call and agent step as a trace, tracks tokens, cost and latency, and runs evaluations that score the outputs. Buyers also search for it as LLM monitoring and observability, or agent observability.
The platforms differ on three decisions. The first is the license: some are open source and free to run yourself, some are open source with limits on offering them as a service, and some are commercial only. The second is who hosts the data, since traces hold prompts and outputs that may contain personal data. The third is the platform it extends: an agent framework, an APM suite you already pay for, an ML platform, or none.
This guide defines the category from the OpenTelemetry project’s own documents, explains what to test, says what replaces an evaluation platform that has been retired, sets out six approaches to buying, names the eight vendors in the table that represent them, and lists published prices. Infrastructure and application monitoring is covered in the APM & Observability Platforms Buyer Guide. Training, tracking and serving your own models is covered in the ML Platforms Buyer Guide.
What Is LLM Observability?
LLM observability applies tracing, metrics and logs to applications that call language models, and adds a step that ordinary monitoring lacks: scoring the output. OpenTelemetry, the open source observability project, sets out what to watch. OpenTelemetry recommends keeping track of how often LLMs are being used, for usage and cost tracking. OpenTelemetry states that latency is important to track since the response time from the model can vary based on the inputs passed to the LLM. OpenTelemetry states that rate limiting is a common challenge, particularly for external LLMs. OpenTelemetry states that tokens directly impact cost and are a measure of response length.
Traces carry the structure. In a retrieval-augmented generation (RAG) application, retrieval and other steps run before and after each model call, and a trace records them in order. An agent multiplies those events: one user request can produce several model calls, retrievals and tool calls, each a span in the same trace. OpenTelemetry defines semantic conventions for GenAI agent and framework operations across both remote services and local frameworks.
Evaluation is what separates the category from APM. OpenTelemetry states that, for an AI agent, given its non-deterministic nature, telemetry is also used as a feedback loop to improve the quality of the agent by using it as input for evaluation tools. In practice that means automated scoring of production traces (by a second model acting as a judge, by code, or by people), test datasets built from traces that went wrong, and experiments that rerun those datasets before each prompt or model change.
Standards decide how easily you can switch. OpenTelemetry states that because observability and evaluation tools for GenAI come from various vendors, it is important to establish standards for the shape of agent telemetry to avoid lock-in caused by vendor or framework specific formats. OpenTelemetry states that its GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository. OpenTelemetry's semantic conventions for generative AI systems carry the status Development. Attribute names can still change, so ask each vendor which version of the conventions it reads and writes.
Which LLM Evaluation Platforms Are Still Sold, and Which Add Guardrails?
One evaluation platform from the previous edition is gone. Patronus AI states on its documentation site that the LLM evaluation platform documented there is retired and no longer offered, and that its evaluation API, SDKs, evaluators, Lynx, GLIDER and Percival are all retired. An open source evaluation tool remains free to run.
Arize describes Phoenix as its open-source, local-first platform for tracing, evaluation, experimentation, and prompt iteration, and tells users to move to Arize AX only when they need it. Arize states that Phoenix can run locally or be self-hosted in your environment.
Runtime guardrails, which score or block a response before a user sees it, are a separate capability from tracing and offline evaluation. Three platforms in the table describe them. Splunk states that Galileo, now Splunk Agent Observability, gives AI teams tools to evaluate and improve agent behavior, track usage and cost, and guardrail actions in real time across 100% of traffic. Weights & Biases states that Weave Guardrails offers pre-built safety scorers, including toxicity, bias, PII detection, and hallucinations. Datadog states that Sensitive Data Scanner is built into Agent Observability at no additional cost. Dedicated guardrail and prompt-injection products are covered in the AI Runtime Security & LLM Guardrails Buyer Guide.
If you used the retired evaluators, list each one and the metric it produced before you choose a replacement. Rebuild the same metrics in the new platform and run them on the same test set, so the scores before and after the switch can be compared.
| If you need | Look at | Why |
|---|---|---|
| Tracing and evaluation you can run yourself at no license cost | Open source platforms | Free to self-host; check the license terms on offering it as a service |
| Runtime guardrails on production traffic | Evaluation-and-guardrail products, or an AI runtime security product | Scores computed before the response is returned, not after |
| A replacement for retired evaluators | Any platform that runs custom LLM-as-a-judge and code evaluators | Rebuild each metric and compare scores on the same test set |
| Monitoring inside the APM suite you already run | APM platform modules | LLM traces next to service and infrastructure data |
What Should You Test in an LLM Observability Platform?
Start with instrumentation. Ask each vendor to trace one of your applications end to end, with every model call, retrieval and tool call as a span, using the framework you already run. Then ask it to ingest the same application instrumented with OpenTelemetry, and compare the two traces.
Check what the trace stores. OpenTelemetry states that model instructions, user messages, and model outputs are considered sensitive and are often large in size, and that its instrumentations should not capture them by default but should provide an option for users to opt in. For production environments where telemetry volume is a concern or sensitive data needs to be handled securely, OpenTelemetry recommends storing content externally and recording references on the spans. Ask each vendor whether it redacts personal data before storage, where content is stored, and who in your organization can read it.
Test evaluation on your own failures. Give each vendor twenty production traces that went wrong. Ask it to turn them into a dataset, write an evaluator that catches the failure, run it as an experiment against a changed prompt, and then run the same evaluator on live traffic at a sampling rate you choose.
Measure what users feel. OpenTelemetry defines a gen_ai.server.time_to_first_token metric for the time a model server spends to generate the first token of a response, which helps measure the time spent in the queue and the prefill phase. Ask whether the platform shows time to first token, total latency, tokens and cost per user, per feature and per model.
Check retention and export last. Retention decides how far back you can investigate a complaint, and it is often priced separately. Ask how traces, scores and datasets are exported, in which format, and whether an export includes the evaluator definitions.
| Capability | What it does | Buyer translation |
|---|---|---|
| Tracing | Records each model call, retrieval, tool call and agent step as a span | Can you find the step where a bad answer started? |
| OpenTelemetry ingest | Accepts spans that follow the GenAI semantic conventions | Can you change vendor without re-instrumenting? |
| Online evaluation | Scores sampled production traces automatically | Do you learn about a quality drop before customers report it? |
| Datasets and experiments | Turns traces into test sets and reruns them on each change | Can a prompt change be tested before release? |
| Human review | Queues traces for people to label | Can subject experts correct the automated judge? |
| Cost and latency | Tracks tokens, cost and time to first token by model and feature | Which feature or customer drives the bill? |
| Data controls | Redaction, retention, regions and self-hosting | Can prompts containing personal data be stored at all? |
Vendors in LLM Observability & Evaluation
Representative vendors include Arize Phoenix, Braintrust, Comet Opik, Datadog LLM Observability, Galileo (Splunk Agent Observability), Langfuse (ClickHouse), LangSmith and Weights & Biases Weave (CoreWeave Agent Lens). The list is alphabetical, is not a ranking, and is not for sale. How we choose vendors
The approaches below follow two questions: who may run the software, and which platform it extends. Two approaches have two platforms each.
A framework vendor’s platform is built by the company behind the agent framework. LangChain describes LangSmith as its commercial agent engineering platform. LangChain states that LangSmith works with any LLM framework, including applications built with the OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or custom implementations, not just LangChain. LangChain states that LangSmith can send its trace data to OpenTelemetry pipelines or ingest OTel data. LangChain states that self-hosted LangSmith is an add-on to the Enterprise plan.
An open source platform under a permissive license can be run in-house without a commercial agreement. Langfuse states that ClickHouse has acquired Langfuse. Langfuse states that it stays open source and self-hostable, with no planned changes to licensing. The license for the Langfuse Software names ClickHouse, Inc. as copyright holder, and makes content outside the listed ee/ directories and third-party components available under the MIT Expat license. Langfuse states that it can be self-hosted using Docker on your own infrastructure, and that some add-on features require a license key. Langfuse states that it captures traces through native SDKs for Python and JS, 100+ library and framework integrations, OpenTelemetry, or an LLM gateway such as LiteLLM, and that it is based on OpenTelemetry.
Open source with a managed platform from the same company pairs a free project with a paid service. Arize's Phoenix page describes Phoenix as self-hostable on your own infrastructure, ELv2 licensed, and built in the open. The Elastic License 2.0 (ELv2) does not allow providing the software to third parties as a hosted or managed service that gives users access to any substantial set of its features or functionality. That limit matters only if you plan to offer the software to others. Arize states that Phoenix accepts traces over OpenTelemetry (OTLP) and provides auto-instrumentation for frameworks such as LlamaIndex, LangChain, DSPy, Mastra and the Vercel AI SDK. Comet states that open source Opik is $0, with the full AI observability and agent testing feature set and the same codebase as the hosted versions. Comet states that Opik online evaluation rules automatically score incoming traces, and that feedback scores, latency, cost and error rates are monitored from the project dashboard.
An evaluation-first platform starts from datasets and experiments and adds production tracing. Braintrust's site says to define what good looks like before you ship: run experiments against real datasets, compare prompts and models side-by-side, and score outputs with LLMs, code, or humans. Braintrust states that production traces can be turned into eval datasets with one click, and that regression tests can be built from real failures and edge cases. Braintrust states that self-hosting is a legacy deployment option and that, starting November 4, 2026, it is not available to new customers. Braintrust points customers who want to keep data in their own cloud to bring your own cloud (BYOC).
APM vendors sell LLM and agent observability as a module beside their infrastructure and application monitoring. Datadog states that Datadog Agent Observability helps engineering teams evaluate, improve, and trace AI agents across development and production in one platform. Datadog states that Agent Observability is a standalone product and can be purchased without any other Datadog subscription. Datadog states that applications can be instrumented in Python, Node.js, or Java, and that teams can bring any custom environment in via OpenTelemetry or the HTTP API. Splunk states that Cisco has completed its acquisition of Galileo. Galileo describes itself as an AI observability and eval engineering platform where offline evals become production guardrails. Galileo states that optimized evals can be distilled into Luna models that monitor 100% of traffic at 96% lower cost.
An ML platform extension adds agent tracing to an experiment tracking product. CoreWeave states that Weave agent tracing workflows are also available in CoreWeave Agent Lens, the successor to Weights & Biases Weave. CoreWeave states that using the Weave library starts with a CoreWeave Forge account. CoreWeave states that an OTel-instrumented agent can send traces to the Weave OTLP endpoint with no Weave SDK installation required. The ML Platforms Buyer Guide covers that platform as a whole.
Two more vendors appear in searches for this category. An AI gateway sits between the application and the model providers and logs every request that passes through it. Helicone states that every AI model can be accessed through the OpenAI SDK, and that providers are switched by changing the model name. Helicone states that its AI Gateway can be self-hosted in your own infrastructure through single-click deployment or a Helm chart. Helicone states that it is no longer part of Mintlify, with details to come. The previous edition also evaluated a platform that has since been retired, as described above. Patronus AI describes itself as a research lab developing training data and simulated environments for frontier AI.
| Vendor | Approach | Where it fits |
|---|---|---|
| Datadog LLM Observability | APM platform module | Sold as Agent Observability; billed per LLM span; no other Datadog product needed |
| Galileo (Splunk Agent Observability) | APM platform module | Evaluations that run as guardrails; acquired by Cisco, now part of Splunk |
| Braintrust | Evaluation-first platform | Experiments and datasets from production traces; BYOC instead of self-hosting |
| LangSmith | Framework vendor’s platform | From LangChain; any framework; self-hosting is an Enterprise add-on |
| Weights & Biases Weave (CoreWeave Agent Lens) | ML platform extension | Agent tracing with guardrail scorers; a CoreWeave Forge account |
| Arize Phoenix | Open source with a managed platform | ELv2; OpenTelemetry and OpenInference; Arize AX as the paid platform |
| Comet Opik | Open source with a managed platform | Open source edition with the hosted feature set; Pro Cloud by the month |
| Langfuse (ClickHouse) | Open source, permissive license | MIT Expat outside ee/ directories; built on OpenTelemetry; acquired by ClickHouse |
Eight platforms across six approaches; Arize Phoenix and Comet Opik share one, as do Datadog and Galileo. The approaches were written first, and each platform was then chosen from those the previous edition evaluated, with Weights & Biases Weave added after Patronus AI retired its platform. A row in this table cannot be bought. Helicone, named above, and other tracing and gateway products can make their own case in the Spotlight below.
How Is LLM Observability Priced?
Every vendor here prices by volume of telemetry, and each counts it differently: traces, spans, units, processed data or LLM spans. Seats are added by some and not others. Convert each quote to the cost of one month of your own traffic before you compare.
Per trace and per seat. LangChain lists the LangSmith Developer plan at $0 per seat per month for 1 seat with up to 5k base traces a month, and the Plus plan at $39 per seat per month with up to 10k base traces a month, then pay as you go. LangChain states that LangSmith base traces have a retention period of 14 days and extended traces 180 days.
Per unit, span or gigabyte. Langfuse lists its Hobby plan as free with 50k units a month, and its Core plan at $29 a month with 100k units included and additional units at $8/100k units. Arize lists AX Pro at $50 per month with 50k trace spans a month and 30 days of retention. Comet lists Pro Cloud at $19 per month for up to 50 team members, 100k spans per month and 60-day data retention. Braintrust lists a $0 a month plan with 1 GB of processed data and 14-day retention, and a Pro plan at $249 a month.
Per LLM span. Datadog states that it bills only on LLM spans, not on tool spans, retrieval spans, or other workflow spans surrounding the model call. Datadog lists a Free tier with up to 40K LLM spans per month and 15-day retention, and a Pro tier starting at $240 per month on-demand for the first 100K LLM spans, $200 per month month-to-month and $160 per month on an annual commitment.
Self-hosting moves the cost to your own infrastructure: databases, object storage and the people who run them. A request for Galileo’s pricing page (galileo.ai/pricing) is answered by Splunk’s pricing page (splunk.com/en_us/products/pricing.html), so that price is a quote. All prices are US list prices as published when this guide was checked in November 2026.
| Meter | You are charged for | Grows with | Where it goes wrong |
|---|---|---|---|
| Traces | Each end-to-end request, whatever its length | Requests | Long retention billed as a separate trace tier |
| Spans or units | Every step inside a trace, and sometimes each score | Steps per request | Agents that call many tools in one request |
| LLM spans | Model calls only | Model calls per request | Retries and fallbacks that double the calls |
| Processed data | Gigabytes of prompts, outputs and metadata | Prompt and context length | Large retrieved documents logged on every span |
| Seats | Each user of the platform | Team size | Reviewers and product staff added for annotation |
How long does implementation take for LLM Observability & Evaluation?
Most of the work is instrumentation and deciding what to store. Agree with security and privacy owners which content may be captured before the first trace reaches a vendor.
Pick one application. Instrument it with OpenTelemetry or the vendor SDK, with content capture off, and confirm every model call, retrieval and tool call appears as a span with tokens, cost and latency.
Turn on prompt and output capture only where approved, with redaction and a retention period. Decide who can read content, and record the decision.
Build a dataset from real failures, write evaluators for the failures that matter, and agree a pass threshold. Run the dataset in CI on every prompt or model change, and sample production traffic for online scores.
Add applications in waves. Route alerts on cost, latency and score drops to the owning team. When migrating from another platform, run both on the same traffic for two weeks and export the old traces and datasets before the contract ends.
What should you ask vendors about LLM Observability & Evaluation?
Ask each vendor to answer in writing and to demonstrate on one of your applications.
-
Must traces stay inside your own network?Yes Shortlist the open source platforms and any vendor with a bring-your-own-cloud option, and confirm the license allows your use.No Include hosted platforms, and check their data regions and retention.
-
Do you already run an APM suite with an LLM module?Yes Test that module first on one application, against one specialist platform.No Weight evaluation depth and OpenTelemetry ingest over APM correlation.
-
Do you need responses blocked or changed before users see them?Yes Require runtime guardrails, and test their latency on your traffic.No Weight offline experiments and online scoring of sampled traces.