Core Concepts

Latitude’s core concepts: projects, spans, traces, sessions, users, tools, memory, cost, search, behaviors, scores, signals, experiments, evaluations, monitors, and datasets.

Core Concepts

Latitude is organized around a small set of concepts, grouped by the loop you work in: Observe what your agent did, Understand what matters, and Refine with monitoring and tests.

Project
├─ Observe
│  ├─ Sessions
│  │  └─ Traces
│  │     └─ Spans (LLM calls, tool calls, retrieval, and more)
│  ├─ Users
│  ├─ Tools
│  ├─ Memory
│  └─ Cost
├─ Understand
│  ├─ Search and Behaviors over traces
│  ├─ Scores attached to traces
│  ├─ Signals grouped from failed scores, or defined by you
│  └─ Experiments comparing slices of traffic side by side
└─ Refine
   ├─ Evaluations that score live traffic
   ├─ Monitors that alert on signals, searches, or raw traffic
   └─ Datasets, local simulations, and regression tests that lock a fix in CI

Projects

A project is the main workspace for one agent, application, or AI feature. Projects keep production data and reliability workflows scoped to the right product boundary.

Project-scoped data includes:

• spans, traces, and sessions
• users, tools, memory stores, and cost analytics
• search, saved searches, and behaviors
• scores from annotations, evaluations, flaggers, and custom checks
• signals grouped from failed scores
• experiments, datasets, evaluations, and monitors
• project settings such as monitoring behavior

Projects belong to organizations. Organization membership, API keys, and settings control who can access a project and how defaults cascade.

Spans

A span is the smallest unit of captured work. It represents one operation inside an agent run, such as an LLM call, tool call, retrieval step, HTTP request, or custom instrumented operation.

Spans can capture:

• input and output content when available
• start time, end time, and duration
• success or error status
• tags and metadata
• provider, model, token usage, and cost for LLM calls

Use spans when you need to understand exactly what happened inside a trace.

Traces

A trace is one complete interaction from start to finish, composed of one or more spans. For a chat agent, a trace commonly represents a single user turn and the agent work needed to respond.

Traces are the main unit used across Latitude:

• the trace detail view shows the full execution path
• search returns matching traces
• annotations, scores, and evaluations attach to traces
• flaggers run on completed traces
• failed scores on traces feed signal discovery

Latitude waits for a trace to stop receiving new spans before treating it as complete. Downstream workflows such as search indexing, flaggers, evaluations, scores, and signal discovery run on completed traces so they do not act on partial executions.

Sessions

A session groups related traces into a multi-turn conversation or workflow. Traces always exist; sessions exist when your application sends a stable session id with telemetry.

Use sessions when multiple traces belong to the same user conversation, support ticket, workflow, or agent task. For example, each chat turn can be one trace while the whole conversation shares one session.

Users

A user is an end user of your agent, identified by the userId you send with telemetry. The Users view groups everything one user did: their sessions, errors, cost, and activity over time, along with the signals affecting them. Use it to see who is impacted by a problem and who is most active.

Tools

A tool is a function your agent can call. Latitude discovers every tool from your telemetry and tracks per-tool usage, error rate, and latency in the Tools view, so you can see which tools are heavily used, failing, or never called.

Memory

Memory is the persistent state your agent reads and writes across interactions. Latitude records every memory operation as a span and versions each record like a commit, so the Memory view shows each store’s current contents, its full change history with diffs, and the session behind every change. See Memory.

Cost

The Cost view breaks down what your agent spends: the spend trend over time, a per-model cost breakdown, cache economics with per-model break-even points and recommendations, and cost per session. Use it to see where the money goes and what caching is actually saving you.

Search

Search finds traces by meaning, exact text, metadata filters, or a combination of all three. It is the discovery layer for behaviors your team wants to inspect, review, and track. Saved searches let teams bookmark useful cohorts and review them repeatedly.

Use search to find traces such as:

• users showing frustration
• failed payments or onboarding problems
• tool loops and tool call errors
• hallucinated or incomplete answers
• traces from a specific user, model, release, environment, tag, or metadata value

Behaviors

A behavior is a topic Latitude automatically discovers by clustering your sessions by meaning. Behaviors surface what your users are actually doing, with trends and outcome metrics, so you find patterns you did not know to search for. Where search starts from a query, behaviors start from a map of your traffic.

Scores

A score is Latitude’s common verdict model for trace quality. Scores can come from human annotations, built-in flaggers, automated evaluations, or custom checks submitted through the API.

Scores answer questions such as:

• Did this trace pass or fail a quality check?
• What feedback explains the verdict?
• Which source produced the verdict?
• Should this failure contribute to signal discovery?

Failed, eligible scores are the primary input for signal discovery. Scores also power analytics and help teams compare automated judgments with human feedback.

Signals

A signal is a recurring failure pattern Latitude tracks. Instead of leaving failures as isolated examples, Latitude groups similar failed scores into named signals with example traces, trends, affected users, lifecycle states, and linked evaluations. You can also define a signal yourself when you already know the behavior to track. See Create a signal.

Signals help teams:

• triage new and escalating production problems
• inspect representative traces for the failure pattern
• decide whether a pattern is real, noise, resolved, or ignored
• generate evaluations that monitor the same failure mode on future traces
• catch regressions after a fix ships

Signal states are new, escalating, ongoing, resolved, regressed, and ignored. A signal can hold more than one at once, such as new and escalating, and a resolved signal that starts occurring again is reopened as regressed. See the signal lifecycle.

Experiments

An experiment compares two or more slices of your traffic side by side. Each slice is a variant defined by filters, an optional search query, and a time range; one variant is the baseline, and every metric Latitude tracks is compared against it. Use experiments to measure how a release, a model swap, or a user cohort differs, rather than eyeballing two dashboards. See Experiments.

Evaluations

An evaluation is an automated monitor that checks traces for a specific behavior or quality criterion. Evaluations run on completed traces and produce scores, so their results feed the same analytics and signal discovery workflows as annotations, flaggers, and custom checks.

Evaluations help teams:

• monitor known failure modes on live traffic
• measure whether a fix reduced or eliminated a behavior
• catch regressions after a signal is resolved
• turn important signals into ongoing monitors
• compare automated judgments with human annotations through alignment metrics

Latitude can generate evaluations from signals, then use human feedback to keep those evaluations aligned with the team’s judgment.

Monitors

A monitor watches a target, such as a signal, a saved search, a tool, or your raw traffic, and opens an incident when a condition is met. When an incident opens, Latitude notifies you in-app, by email, or in Slack. Every project comes with system monitors that watch the signal lifecycle, and you can create your own.

How the concepts fit together

1. A project receives telemetry from your agent.
2. Agent operations arrive as spans.
3. Related spans form completed traces, and related traces group into sessions.
4. Search and Behaviors help you find the traces and topics that matter; Users, Tools, Memory, and Cost break the same activity down by end user, tool, memory store, and spend.
5. Annotations, flaggers, evaluations, and custom checks create scores on traces.
6. Failed scores cluster into signals you can triage, monitor, and fix, and you can also define a signal yourself. Experiments measure whether a change moved the metrics.
7. Monitors watch signals (and searches, tools, or raw traffic) and open incidents that alert you when something needs attention.
8. Datasets built from real traces feed local simulations and regression tests, so a fix is checked against the same cases before it ships; if the failure returns in production anyway, the resolved signal reopens as regressed.

Next steps

• Start tracing: send your first spans and traces to Latitude.
• Observability: inspect traces, sessions, users, tools, and memory.
• Search and Behaviors: find the traffic that matters.
• Signals: understand how failed scores become tracked signals.
• Monitors: get alerted when something needs attention.
• Simulate agents locally: run a dataset against your agent in TypeScript or Python.