Why Rust Tracing and Observability Tools Make Debugging Robots Easy

This blog post is adapted from a talk I gave for the “Rust for Robotics” workshop at IROS 2026, in Pittsburgh, PA. The prevailing theme of the day was an extended discussion about why we choose Rust as the language we use for programming in Robotics and Autonomy.

In particular, the talk I presented pertained to tracing and observability tools in Rust. The thesis of the talk was straightforward: Rust’s ecosystem is a natural fit for Robotics and Autonomy because it provides a number of features and mechanisms for better understanding what our software is doing in a production environment.

Tracing, Observability, and Debugging

Before getting too deep into the technical details of how we use Rust’s tracing ecosystem at Tangram, it is important to understand some of the terminology we will use throughout. For a lot of programmers in robotics & autonomy today these may seem obvious; however, there’s a wealth of tools that treat “tracing” as just one technology, or as just some form of “fancy logging.” To get started with this topic, I want to present the following definitions:

• Tracing (structured logging): understanding the order and flow of execution through the recording of structured objects (i.e., not just strings of text).

• Observability: the ease of which one can observe behaviors in a system. Explicitly, we want to separate how we view this for the purposes of this post from the notion of mathematical observability, which is a more strict definition we have previously referenced on our blog.

• Debugging/Debugger: A debugger or debugging tool is any tool that helps you better understand your software, or in the case of MetriCal, your calibration. This is a much broader definition than what we might think of when we think of traditional software “debuggers” (e.g., gdb, lldb), but the focus here is to look at how Rust and its tracing ecosystem act as the basis for building excellent debugging behavior into our own programs!

When working on any sophisticated robotics & autonomy stacks today, it is impossible to understate how much raw complexity is inherent in just about any piece of the problem. What matters most, and the differentiating factor between useful systems and great systems, is whether or not they help you break down and understand this complexity into something actionable. In the age of LLMs and Physical AI it is no longer enough to just have one part of your robot that “works,” if it is not accompanied by an understanding of how it works and how to integrate that from top to bottom in your organization.

Why Rust? Why tracing?

Rust has a lot of nice properties that make it easy to understand a system both locally, and within the broader context of your autonomy or perception system. For example:

• Algebraic types allow you to understand API boundaries and valid versus invalid (hopefully unrepresentable!) states through typestate analysis

• Marker traits like Send and Sync inform programmers how a given abstraction will behave in concurrent environments

• and more!

Beyond Rust on its own, the tracing crate and its ecosystem can not only allow you to extend your reasoning about local properties to other parts of the system, but likewise allow you to evaluate and introspect how your Rust program behaves in the context of a much larger system of programs and protocols that may not exist solely within the Rust ecosystem!

Contrasts with Other Tooling

To make this argument that tracing and in particular structured logging is one of the best features of using Rust for your robotics & autonomy, I want to contrast with some existing tools in the observability and debugging space that might be used today. Without providing an exhaustive list, some of the most common tools in this space for debugging or observability are:

• perf, on Linux

• eBPF or bpftrace on Linux

• dtrace

• gdb/lldb

Many of these focus on being completely agnostic to the program in question (e.g., gdb), or focusing on your entire system as a whole (e.g., perf). Most of these “traditional” tools have the unfortunate shortcomings of:

1. Needing to be installed in production

2. Requiring root-level permissions on your robot, or attached debug info in your compiled program

3. Guarantees that they are actually running when something goes wrong, or at least when events happen that you want to observe

The advantage of these systems is that they can mostly work wherever they are deployed; conversely, their disadvantages stem from being able to effectively deploy these in production, on real robots in the field.

Tracing in Rust, on the other hand, is useful when you want to approach this problem from the other end. In contrast to the other tools, tracing is a library which means you are annotating or instrumenting your code directly. This means adding lines of code and recompiling your software, but some advantages are:

• You do not need any special elevated permissions to run tracing compared to whatever your software uses or needs today

• Tracing often gets initialized when your program does, so it should always be running if your program is running

• The values structured values you record in tracing retain the structure you give it. So you can often build subscribers that can understand and work with existing types, abstractions, and idioms that you already use!

For these reasons, tracing and Rust in particular can be an excellent fit for writing robotics software. Being able to break down complexity and understand what is running on a device that may not be in your physical proximity or direct control is hugely beneficial.

Tracing Basics

Tracing in Rust gives us some basic primitives to start asking questions that build a better understanding our software and how it is executing. Fortunately, there’s only three concepts that one needs to worry about:

1. Spans

2. Events

3. Subscribers

To explain each of these, we provide some examples of where Tangram uses these directly today.

Spans

Spans represent a period or duration in time. Spans do not themselves record any structure used in the program, but give us a sense of when something begins and ends.

One example where we use spans at Tangram today is in building our AutoCal product. We sometimes run multiple nonlinear optimizations back to back, with different M-estimation settings for down-weighting outliers. Spans give us one way of understanding how long this takes in practice, and we have used it to better understand how a feature like M-estimation in our solver affects performance when run back-to-back.

let l2_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-l2");
let l2_result = l2_span.in_scope(|| solver.solve(problem, &config, None))?;

problem.set_m_estimators(Insha::default().into());

let insha_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-insha");
let insha_result =
    insha_span.in_scope(|| solver.solve(problem, &config, l2_result))?;

let l2_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-l2");
let l2_result = l2_span.in_scope(|| solver.solve(problem, &config, None))?;

problem.set_m_estimators(Insha::default().into());

let insha_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-insha");
let insha_result =
    insha_span.in_scope(|| solver.solve(problem, &config, l2_result))?;

let l2_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-l2");
let l2_result = l2_span.in_scope(|| solver.solve(problem, &config, None))?;

problem.set_m_estimators(Insha::default().into());

let insha_span = tracing::span!(tracing::Level::DEBUG, "autocal-solve-insha");
let insha_result =
    insha_span.in_scope(|| solver.solve(problem, &config, l2_result))?;

It is not super important that you understand the specifics of optimization or how our solver works to get a picture of what a span is for. Specifically, the point to focus on is that we have two spans, each with a unique name, that will log a structure with entry/exit timestamps. This allows us to directly represent how long a given operation takes in our system.

One interesting, but non-obvious, part of how spans are represented is that they carry some information about their parent spans. Later, we will demonstrate how some subscribers in the tracing ecosystem use this data to construct flame charts or stack-like representation of spans.

The tracing::instrument Attribute Macro

In my original talk, I also detailed how this same code can be instrumented automatically by using the tracing::instrument attribute macro. This attribute can be effectively leveraged so that all future callers of e.g., solver.solve as shown above produce spans as well. The way this works is to add the attribute to the function in question:

/// Solves the least-squares problem
#[tracing::instrument(
    level = "debug",
    skip_all,
    fields(loss = problem.loss_kind()),
)]
pub fn solve<Problem, P>(
    &self,
    problem: Problem,
    config: &LeastSquaresSolverConfig,
    initial_parameters: Option<P>,
) -> Result<Solution<P>, Error> {
    // ...
}

/// Solves the least-squares problem
#[tracing::instrument(
    level = "debug",
    skip_all,
    fields(loss = problem.loss_kind()),
)]
pub fn solve<Problem, P>(
    &self,
    problem: Problem,
    config: &LeastSquaresSolverConfig,
    initial_parameters: Option<P>,
) -> Result<Solution<P>, Error> {
    // ...
}

/// Solves the least-squares problem
#[tracing::instrument(
    level = "debug",
    skip_all,
    fields(loss = problem.loss_kind()),
)]
pub fn solve<Problem, P>(
    &self,
    problem: Problem,
    config: &LeastSquaresSolverConfig,
    initial_parameters: Option<P>,
) -> Result<Solution<P>, Error> {
    // ...
}

Which changes the code that I shared previously to:

let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

let insha_result = solver.solve(problem, &config, l2_result)?;

let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

let insha_result = solver.solve(problem, &config, l2_result)?;

let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

let insha_result = solver.solve(problem, &config, l2_result)?;

The ability to add these macros does embed a kind of domain-specific language, but is one of the most compelling aspects of making tracing in Rust feel like a “natural” part of the language.

Events

Events, unlike spans, describe instants in the software. If you are familiar with any bog-standard logging library, these might seem familiar. Using our previous example, we could uniquely describe an “instant” as:

tracing::debug!(loss = SquaredLoss::default(), "Solving with squared loss");
let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

tracing::debug!(loss = Insha::default(), "Solving with Insha loss");
let insha_result = solver.solve(problem, &config, l2_result)?;

tracing::debug!(loss = SquaredLoss::default(), "Solving with squared loss");
let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

tracing::debug!(loss = Insha::default(), "Solving with Insha loss");
let insha_result = solver.solve(problem, &config, l2_result)?;

tracing::debug!(loss = SquaredLoss::default(), "Solving with squared loss");
let l2_result = solver.solve(problem, &config, None)?;

problem.set_m_estimators(Insha::default().into());

tracing::debug!(loss = Insha::default(), "Solving with Insha loss");
let insha_result = solver.solve(problem, &config, l2_result)?;

Events can be nested within spans, and similarly carry information about what span they were executed in. This can make it easy to later query this data in such a way as to identify the surrounding context of a given event.

Subscribers

The most interesting and user-facing of the three components of the tracing ecosystem are subscribers. These are anything we might use to consume, display, or process span and event data. Some examples could be:

• stderr logs

• JSON logs

• flame graphs/flame charts

• a database of events

In fact, if you have used MetriCal, you’ve certainly seen our HTML reporting. The backbone of this is collecting events that get emitted both to stderr in your terminal and our reporting subscriber, and transparently producing a log of the software that can later be viewed via your browser. This way, the report and how it showed up in your terminal are a shareable artifact you can save, send, and show others directly, without having to install any fancy terminal capture or effects on your production line or robot.

Another excellent subscriber is tracing-chrome, which emits Perfetto-compatible JSON traces. In practice, this looks like some kind of trace.json file on your filesystem, but can be directly read in via Perfetto. From this, you can get a wealth of information about your system. For example:

Using a trace like the above, we can look at either the flame chart representation in the middle of the screen, or look at the flame graph in the current selection window on the bottom. This is an example of our camera initialization as profiled using the existing tracing tools. Each of those “rows” on the left side of the flame chart represents the work a given thread is doing, and you can see how work gets distributed in your system!

Ultimately, this provides a huge opportunity for being able to query, display, and interpret events in your software. By leveraging tracing and Rust, we’re able to support multiple subscribers in our software with relatively small overhead both in terms of the executable performance and the amount of code that is needed to instrument or annotate spans and events in our code.

:::tip Merging and Querying Perfetto Traces

One other neat feature of using Perfetto this way in more complex autonomous systems: if you have multiple traces from multiple Rust programs, you can actually merge these traces together and start tying events in one program to delays or events in other software.

The Python-based trace processor, while not Rust itself, is an excellent way to convert your tracing output into an SQL-like format that can be queried in a programmatic way. We have been experimenting with ways to use LLMs as a way to construct queries that match some question we might have about how our software runs in production.

Conclusion

Structured logging, and specifically the tracing ecosystem, is an extremely compelling reason to build your Robotics & Autonomy software in Rust. Here at Tangram, we have uniquely leveraged this to not only build better debugging and observability for “the software,” but hopefully demonstrated some ways in which we use this tooling to build out ways in which we can debug calibrations in our system. Whether through our HTML reporting, or a more in-depth performance trace of our system, Rust uniquely positions us to observe and debug the behavior of our software with an ease that other programming languages just do not offer today.