Observability

Observability is how well you can understand what your application is doing from the outside, without adding ad-hoc debug code everywhere. For LLM-assisted systems this matters even more than for traditional software: the model is non-deterministic, so tracking things like token usage, latency, error rates, tool calls, and model drift is what lets you treat an unpredictable system as reliable.

Before you start

• Build and run an agent — spans and logs are emitted from completion and agent calls.
• Set up a provider client (e.g. OpenAI) so the examples below have something to trace.
• Have RUST_LOG and cargo run available; the basic setup relies on both.

Rig is minimal and unopinionated here. Internally it uses the tracing crate to emit logs and spans, which you can consume with any tracing-subscriber layer, log facade, or OpenTelemetry exporter you like.

What observability gives you in Rig

Rig emits structured telemetry that lets you:

• Inspect model prompts and responses
• Understand agent behaviour across multiple turns
• See tool calls and their outputs
• Debug streaming vs non-streaming completions
• Compare latency and behaviour over time

Rig follows the OpenTelemetry GenAI Semantic Conventions, which makes it compatible with modern GenAI observability platforms such as Langfuse, Arize Phoenix, and any other OpenTelemetry-compatible backend.

Instrumentation levels

Rig uses two logging conventions:

• INFO level: spans marking the start and end of operations
• TRACE level: detailed request/response message logs for debugging

Basic setup with tracing-subscriber

The simplest way to get started is with tracing-subscriber’s formatting layer:

tracing_subscriber::fmt().init();

This requires you to set RUST_LOG every time you want to see logs, for example:

RUST_LOG=trace cargo run

That’s fine for a one-off, but not for quick iterative loops. Combine it with an EnvFilter to set a default level automatically — here info for general logging and trace specifically for Rig:

tracing_subscriber::registry()
    .with(
        tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| "info,rig=trace".into()),
    )
    .with(tracing_subscriber::fmt::layer())
    .init();

To attach your own spans and events, use the #[tracing::instrument] macro. It automatically enters a span (a monitored unit of work) and any events you emit are recorded under it:

use rig::{
    client::{CompletionClient, ProviderClient},
    completion::Prompt,
    providers::openai,
};
use tracing::{info, instrument};

#[instrument(name = "process_user_query")]
pub async fn process_query(user_input: &str) -> Result<String, Box<dyn std::error::Error>> {
    info!("Processing user query");

    let openai_client = openai::Client::from_env()?;

    let agent = openai_client
        .agent("gpt-5.5")
        .preamble("You are a helpful assistant.")
        .build();

    // This completion call emits spans automatically.
    let response = agent.prompt(user_input).await?;

    info!("Query processed successfully");
    Ok(response)
}

Driving it from main:

use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::registry()
        .with(
            tracing_subscriber::EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "info,rig=trace".into()),
        )
        .with(tracing_subscriber::fmt::layer())
        .init();

    let response = process_query("Hello world!").await?;
    println!("Response: {response}");

    Ok(())
}

INFO process_user_query: Processing user query
INFO process_user_query:invoke_agent{gen_ai.operation.name="chat" gen_ai.system="openai"}: new
TRACE process_user_query:invoke_agent: request: {"model":"gpt-5.5","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello world!"}]}
TRACE process_user_query:invoke_agent: response: {"choices":[{"message":{"role":"assistant","content":"Hello! How can I help you today?"}}],"usage":{"prompt_tokens":21,"completion_tokens":9}}
INFO process_user_query:invoke_agent: close time.busy=812ms
INFO process_user_query: Query processed successfully
Response: Hello! How can I help you today?

Running this you’ll see INFO spans for the completion lifecycle, TRACE logs with the actual request/response payloads, and your own process_user_query span.

JSON logging

For production, tracing can emit structured JSON via .json() on the fmt layer:

use tracing_subscriber::{fmt, layer::SubscriberExt, util::SubscriberInitExt};

tracing_subscriber::registry()
    .with(
        tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| "info,rig=trace".into()),
    )
    .with(fmt::layer().json())
    .init();

This is handy when logs are shipped to a log drain, since you can query them with jq:

tail -f <logfile> | jq .

Quickstart with Langfuse (no collector required)

If you just want to see what Rig is doing, the fastest path is Langfuse. Rig works out-of-the-box with Langfuse over OpenTelemetry, and you can integrate it without running an OpenTelemetry Collector — ideal for local development and most production workloads.

Add the dependencies:

[dependencies]
opentelemetry = "0.31"
opentelemetry_langfuse = "0.6"
tracing-opentelemetry = "0.31"
tracing-subscriber = "0.3"

Then initialise tracing:

use opentelemetry_langfuse::LangfuseTracer;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

fn init_tracing() {
    let langfuse_public_key =
        std::env::var("LANGFUSE_PUBLIC_KEY").expect("LANGFUSE_PUBLIC_KEY not set");
    let langfuse_secret_key =
        std::env::var("LANGFUSE_SECRET_KEY").expect("LANGFUSE_SECRET_KEY not set");

    let tracer = LangfuseTracer::builder()
        .with_public_key(langfuse_public_key)
        .with_secret_key(langfuse_secret_key)
        .with_host("https://cloud.langfuse.com")
        .build()
        .expect("failed to create Langfuse tracer");

    tracing_subscriber::registry()
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(tracing_subscriber::fmt::layer())
        .init();
}

Once this is set up, model calls, agent invocations, multi-turn agent loops, and tool executions automatically appear in the Langfuse UI.

Agent span naming and customisation

By default, agent spans use a generic name such as invoke_agent. This is a limitation of tracing: span names can’t be changed after creation.

However, Rig attaches attributes like gen_ai.agent.name and gen_ai.operation.name. You can use these to rename spans in your observability backend (see the collector transform processor below).

Exporting to an OpenTelemetry Collector (advanced)

You only need an OpenTelemetry Collector if you want to forward telemetry to multiple backends, you already run OTel infrastructure, or you need custom processing or redaction. The collector approach is more flexible than a vendor-specific subscriber layer, at the cost of some setup.

Dependencies

[dependencies]
# `cargo add rig` (imported in code as `rig`)
rig = "0.39.0"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-opentelemetry = "0.31"
opentelemetry = { version = "0.30", features = ["trace"] }
opentelemetry_sdk = { version = "0.30", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "0.30", features = ["tonic", "trace"] }

Wiring up the exporter

Build an OTLP span exporter, wrap it in a tracing-opentelemetry layer, and stack it with a filter and a fmt layer so traces go to both stdout and the collector:

use opentelemetry::trace::TracerProvider;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{Resource, trace::SdkTracerProvider};
use tracing::Level;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .with_protocol(opentelemetry_otlp::Protocol::HttpBinary)
        .build()?;

    let provider = SdkTracerProvider::builder()
        .with_batch_exporter(exporter)
        .with_resource(Resource::builder().with_service_name("rig-service").build())
        .build();
    let tracer = provider.tracer("example");

    let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);
    let filter_layer = tracing_subscriber::filter::EnvFilter::builder()
        .with_default_directive(Level::INFO.into())
        .from_env_lossy();
    let fmt_layer = tracing_subscriber::fmt::layer().pretty();

    tracing_subscriber::registry()
        .with(filter_layer)
        .with(fmt_layer)
        .with(otel_layer)
        .init();

    let response = process_query("Hello world!").await?;
    println!("Response: {response}");

    // Flush and shut down the tracer provider on exit.
    let _ = provider.shutdown();

    Ok(())
}

Collector configuration

The collector receives OTLP traces, optionally transforms them, and exports them to your backend. This example renames invoke_agent spans using the gen_ai.agent.name attribute and forwards to Langfuse:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  transform:
    trace_statements:
      - context: span
        statements:
          # Rename the span if it's "invoke_agent" and has an agent name attribute
          - set(name, attributes["gen_ai.agent.name"]) where name == "invoke_agent" and attributes["gen_ai.agent.name"] != nil

exporters:
  debug:
    verbosity: detailed
  otlphttp/langfuse:
    endpoint: "https://cloud.langfuse.com/api/public/otel"
    headers:
      Authorization: "Basic ${AUTH_STRING}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform]
      exporters: [otlphttp/langfuse, debug]

Note

Langfuse uses HTTP basic auth in the form username:password, where the username is your Langfuse public key and the password is your Langfuse secret key. Base64-encode that pair to produce AUTH_STRING.

To run the collector, build a Dockerfile from the OpenTelemetry Collector Contrib image with your config baked in:

# Start from the official OpenTelemetry Collector Contrib image
FROM otel/opentelemetry-collector-contrib:0.135.0

# Copy your local config into the container
# Replace `config.yaml` with your actual filename if different
COPY ./config.yaml /etc/otelcol-contrib/config.yaml

Build it with docker build -t <some-tag-name> -f <dockerfile-filename> ..

Logs only, no spans

Because Rig uses tracing, you can ignore spans entirely and print only log messages. The following tracing_subscriber::Layer prints the level and message and nothing else:

use std::io::Write;

#[derive(Clone)]
struct MessageOnlyLayer;

impl<S> tracing_subscriber::Layer<S> for MessageOnlyLayer
where
    S: tracing::Subscriber + for<'a> tracing_subscriber::registry::LookupSpan<'a>,
{
    fn on_event(&self, event: &tracing::Event<'_>, _ctx: tracing_subscriber::layer::Context<'_, S>) {
        use tracing::field::{Field, Visit};

        struct MessageVisitor {
            message: Option<String>,
        }

        impl Visit for MessageVisitor {
            fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
                if field.name() == "message" {
                    self.message = Some(format!("{:?}", value));
                }
            }
        }

        let mut visitor = MessageVisitor { message: None };
        event.record(&mut visitor);

        if let Some(msg) = visitor.message {
            let msg = msg.trim_matches('"');
            let metadata = event.metadata();

            let level = match *metadata.level() {
                tracing::Level::TRACE => "TRACE",
                tracing::Level::DEBUG => "DEBUG",
                tracing::Level::INFO => "INFO",
                tracing::Level::WARN => "WARN",
                tracing::Level::ERROR => "ERROR",
            };

            let _ = writeln!(std::io::stdout(), "{level} {msg}");
        }
    }
}

Wire it up like any other layer:

use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

tracing_subscriber::registry()
    .with(tracing_subscriber::EnvFilter::new("info"))
    .with(MessageOnlyLayer)
    .init();

Provider-specific integrations

Rig doesn’t ship pre-built subscriber layers for specific vendors — the collector approach above sends traces to any observability platform, which is more flexible. If you’d like a first-class layer for a specific integration, open a feature request on the Rig GitHub repo.

Troubleshooting

• Spans appear out of order. If an operation completes in under ~1ms, some backends may display spans slightly out of order due to timestamp resolution. This is usually not an issue in real production workloads.

See also

• Complete examples: basic and OpenTelemetry binaries, plus the collector YAML and Dockerfile
• tracing and tracing-subscriber documentation
• OpenTelemetry Rust and the OTEL Collector documentation

Next steps

Error handlingTurn the error rates you now trace into typed, recoverable failures.

EvalsGo from watching runs to scoring them, using the traces as your data.

TestingPin down non-deterministic behaviour with tests around your agents and tools.