Introduction

datavis is a desktop tool for watching streaming telemetry in real time. It pulls in live channels over several transports, decodes them — Protobuf schemas can be picked up on the fly — and draws them in a workspace of panels you arrange yourself. You can record a session and scrub back through it later through the very same panels.

It is written in Rust on top of egui.

What it does

• Reads live data from ZeroMQ (Protobuf), MQTT, WebSocket (InfluxDB line protocol), and organisation-specific transports through an external bridge adapter. Protobuf payloads are decoded at runtime with prost-reflect, so you do not have to compile schemas in ahead of time.
• Arranges panels by carving the window into tiles and dragging channels out of the sidebar onto whichever panel you want. Select a few at once and drop them together.
• Ships a handful of panel types: waveform, numeric readout, gauge, FFT spectrum, XY scatter, state graph, status, and a text log.
• Records to MCAP — including any Protobuf schemas it discovered — and plays the file back through the same panels. Replay and live look identical because panels never know which one they are getting.
• Runs Python scripts (numba-compiled) to derive new channels from existing ones, live and during replay.
• Reads its channels and layout from config.toml. A demo source is built in, so you can poke at it without wiring up anything real.

The core idea

One trait — ChannelStore — is the seam that hides live-vs-replay from everything above it. Panels read samples through that trait and never learn which mode they are in. Recording, playback, and the larger-than-RAM lazy loader all sit behind the same interface. This design choice recurs throughout the guide; when a feature "just works" in both live and replay, this is why.

How to read this guide

• Getting Started gets the app running in under a minute using the demo source.
• Configuration covers config.toml: channels, defaults, and the panel layout.
• Data Sources covers each transport and how to wire it up.
• Visualising covers panels, zoom, and cursors.
• Recording & Playback covers MCAP capture, size-based auto-split, and scrubbing recordings back — including files larger than RAM.
• Scripting covers deriving channels in Python.
• Reference is the terse lookup: CLI flags, every config.toml section, and the internal architecture.

A note on provenance. Large parts of this project — this guide included — were written with an AI assistant. Expect confident-looking code that has not met every edge case. Check anything here you would not take on faith.

Getting Started

Build and run

datavis is a Rust application. From the repository root:

cargo run     # start the app — the demo source is enough to see it working
cargo test    # run the tests

The scripting feature is on by default. It pulls in PyO3/numpy, which need a Python interpreter (python3 + python3-config) on PATH at build time. Build inside the nix develop shell (it provides python3 + numba) or install python3-dev. To build the base app without a Python toolchain:

cargo run --no-default-features

First run: the demo source

You do not need any live inputs to try datavis. The built-in demo source synthesises channels:

cargo run -- --demo

Add --demo-freq <HZ> to change the sine frequency (default 1.0). See the Demo Source page for details.

The config file

Channels and the panel layout both live in config.toml, read from the working directory at startup. If no config.toml is present, datavis offers to save the built-in default, load a different file, or run with defaults just this once.

A fresh config has no channels or panels. You add visualisations from the panel type picker in the UI, and the layout is written back into config.toml automatically as you arrange it. See Configuration.

A minimal live setup

To watch a real ZeroMQ Protobuf stream:

cargo run -- --endpoint tcp://localhost:5555 --schema schema.proto

--schema points at the .proto file describing the wire messages; datavis compiles it at startup. Then declare which proto fields become channels in config.toml (see Channels) and drag them onto panels.

Where to go next

• Wire up a different transport → Data Sources
• Lay out panels → Panels & Layout
• Record and replay a session → Recording to MCAP

Configuration Overview

datavis reads everything it needs from a single config.toml in the working directory. That one file holds several independent sections:

How the file is parsed

Each concern parses the whole config.toml independently and ignores sections it does not own. No parser uses deny_unknown_fields across the whole document, so unrelated sections never cause a parse error. This is why you can drop a [recording] or [scripts] block into the same file the layout parser reads without either stepping on the other.

The practical consequence: sections are optional and order-independent. Omit one and its feature falls back to a sensible default (or turns off).

What is hand-edited vs. GUI-managed

• Hand-edited: [channels.*], [defaults], [[sources.bridge]], [scripts], [recording]. These are stable declarations you write once.
• GUI-managed: the [screens.*] layout. datavis writes the tile arrangement and panel list back to config.toml as you drag things around and on exit. You can hand-edit it, but expect the app to overwrite your formatting.

Top-level keys

default_window_s = 10.0   # default visible time span for time-series panels

default_window_s seeds the time window new time-series panels (waveform, state graph) open with. Individual panels can override it once you zoom.

A complete field-by-field listing of every section lives in the config.toml Reference.

Channels

A channel is a named, typed stream of samples. The [channels."name"] tables map raw source data — a Protobuf field, an MQTT topic, an Influx field — onto channels you can drop on panels. The channel registry built from these tables is what the whole app reasons about; panels resolve channel names to ids through it and never parse names themselves.

A Protobuf channel

For a ZeroMQ/Protobuf source you declare which message field carries the value and which carries the timestamp:

[channels."sensor.acceleration.x"]
topic      = "accel"                       # source topic this field arrives on
proto_path = "AccelBatch.samples.x"        # value field path in the proto
ts_path    = "AccelBatch.samples.t_ns"     # timestamp field path (nanoseconds)
type       = "float"                       # float | int | bool | text
unit       = "m/s²"
color      = "#ff0000"
max_rate   = 100000                        # Hz — used to size the ring buffer
history_s  = 10.0                           # seconds of history retained
eu_scale   = 1.0                            # raw → engineering-unit scale
eu_offset  = 0.0                            # raw → engineering-unit offset

Fields

Ring-buffer capacity is derived as max_rate × history_s × 1.2. If you omit max_rate / history_s, per-channel values fall back to the file-wide [defaults] (or the built-in 1000 Hz / 10.0 s).

Sample types

A state / integer channel

[channels."motor.state"]
topic      = "status"
proto_path = "StatusBatch.samples.state"
ts_path    = "StatusBatch.samples.t_ns"
type       = "int"
max_rate   = 1000
history_s  = 30.0

A text / log channel

[channels."system.log"]
topic      = "log"
proto_path = "LogBatch.samples.message"
ts_path    = "LogBatch.samples.t_ns"
type       = "text"
max_lines  = 500

Engineering-unit scaling

eu_scale and eu_offset convert a raw source value into engineering units on ingest, before it reaches the store: eu = raw × eu_scale + eu_offset. Everything downstream — panels, cursors, recordings — sees the scaled value.

Channels without a proto path

Not every source needs proto_path / ts_path:

• MQTT / WebSocket channels can be discovered at runtime — a topic dragged onto a panel — and never appear in config.toml at all. See MQTT and WebSocket.
• Bridge channels declare only topic + sample_type, because the bridge's payload schema is fixed and built in. See Bridge Adapters.

Constraint: topic names must be unique across coexisting sources. A bridge and a ZeroMQ source cannot both emit topic accel — they share one registry.

Defaults

The optional top-level [defaults] table supplies max_rate and history_s for any channel that omits them. It saves you from repeating the same buffer sizing on every channel when a whole file targets, say, 100 kHz.

[defaults]
max_rate  = 100000
history_s = 5.0

[channels."x"]
type = "float"        # inherits max_rate and history_s from [defaults]

[channels."y"]
type      = "float"
max_rate  = 1000      # a per-channel value always wins over [defaults]

What it governs

[defaults] feeds both statically-configured channels and runtime-discovered ones (an MQTT or WebSocket topic dragged onto a panel), so a whole session can share one buffer-sizing policy.

Ring capacity is still computed the same way — max_rate × history_s × 1.2 — just with the resolved values.

What it does not govern

Only max_rate and history_s are settable in [defaults]. Other fields keep their per-field built-in defaults and cannot be set here:

• color, unit, eu_scale, eu_offset, max_lines

There is a single global table — no per-group or per-pattern defaults.

Precedence

1. A per-channel max_rate / history_s in the channel's own table.
2. Otherwise the value from [defaults].
3. Otherwise the built-in default (1000 Hz / 10.0 s for configured channels).

Omitting [defaults], or omitting either key within it, preserves the built-in behaviour exactly. Either key may appear alone.

Panels & Layout

The panel layout lives in config.toml under [screens.*]. Unlike the channel tables, this section is GUI-managed: datavis writes it back as you rearrange panels and on exit. You can hand-edit it, but the app owns its formatting.

Screens

A screen is one named free-canvas workspace of panels. Multiple screens are switchable from the toolbar. Each screen carries its list of panels, each with its own canvas position:

default_window_s = 10.0

[screens.main]

[[screens.main.panels]]
type = "waveform"
frame = [0.0, 0.0, 0.5, 0.4]
channels = ["load/ch0", "load/ch7", "scripts/rms"]
cursors = false
dots = false

[[screens.main.panels]]
type = "state_graph"
channel = "load/state0"

frame

Each panel carries its canvas geometry as frame = [x, y, w, h], four numbers in 0..1 normalized to the screen area (x/y = top-left corner, w/h = size). Panels are placed freely, never overlap, and scale with the window. Omit frame and datavis auto-arranges the screen's panels into a grid on load, then writes real frames back as you move them.

[[screens.<name>.panels]]

One table per panel. Common keys:

• type — the panel type (waveform, gauge, state_graph, status, spectrum, numeric, xy_scatter, log). See Panel Types.
• channels — a list of channel names, for multi-channel panels (waveform, FFT).
• channel — a single channel name, for single-channel panels (gauge, status, state graph).
• Panel-specific keys — e.g. waveform's cursors and dots, status's states, gauge's min/max. Each panel type documents its own keys.
• style (gauge) — the gauge shape: bar (default, horizontal fill), vertical (bottom-up bar), led (segmented VU meter), or arc (270° dial). Chosen from the panel's settings; omitted from config when bar.
• colors (waveform) — an optional { "channel.name" = "#rrggbb" } table of per-channel line-color overrides. Set from the legend's right-click menu; channels without an entry cycle through the Catppuccin accent palette.

Panels resolve channel names to ids through the channel registry at load. A name that does not resolve, or a channel of the wrong type for the panel, is shown as an inline error — never a crash.

Building a layout in the GUI

1. Add a panel from the toolbar, or right-click empty canvas → Add panel here.
2. Choose a panel type from the picker.
3. Drag a panel by its title bar to move it — or hold Ctrl (⌘ on macOS) and drag anywhere in the panel. Drag the bottom-right grip to resize. Panels snap to each other's edges and sizes and block on contact, so they never overlap.
4. Right-click a panel (title bar or body) for its menu: Settings, Reset zoom, and Remove panel. The title-bar gear and ✕ do the same. The dot in the title bar tracks data freshness — green when samples arrive at the channel's usual cadence, amber when they slow, red when they stop.
5. Drag channels from the sidebar onto a panel. Select several and drop them together onto a multi-channel panel.
6. The arrangement is saved back to config.toml automatically.

Time window

default_window_s (top level) sets the visible span new time-series panels open with. Zooming a panel overrides it locally; see Zooming & Cursors.

Data Sources Overview

datavis ingests live data from several transports. They can run concurrently — enable as many as you need — and all feed the same channel store. The status indicator reads LIVE if any configured source is receiving data.

The common source interface

Every transport implements one DataSource trait, giving them a shared lifecycle: build, spawn onto the store, and report a conn_state. This is why sources compose freely — the app pushes each onto a list and treats them uniformly. The MQTT-family sources (MQTT, WebSocket) additionally share a scalar ingest helper that writes a single decoded value into the store.

Enabling several at once

Nothing stops you running, say, a ZeroMQ source and an MQTT source and a bridge:

cargo run -- \
  --endpoint tcp://localhost:5555 --schema schema.proto \
  --mqtt-endpoint localhost:1883 \
  --ws-listen 127.0.0.1:8086

Bridges are added through config.toml rather than the command line, since a bridge needs a command and arguments, and several may run at once. See Bridge Adapters.

Topic uniqueness. Because all sources share one channel registry, a topic name must be unique across every coexisting source. Two sources cannot both claim topic accel.

Recording is source-agnostic

Recording is available whenever any ingest source is active. Each source feeds the same record queue, and the MCAP writer captures whatever schema each message carries. See Recording to MCAP.

ZeroMQ + Protobuf

The ZeroMQ source is datavis's default live transport. It subscribes to a ZMQ SUB endpoint and decodes each message against a Protobuf schema you supply.

Enabling it

cargo run -- --endpoint tcp://localhost:5555 --schema schema.proto

• --endpoint <ADDR> — the ZMQ SUB endpoint. Default tcp://localhost:5555.
• --schema <PATH> — the .proto file describing the wire messages. Default schema.proto.

datavis compiles the .proto at startup (via protox) into a descriptor set and decodes incoming messages reflectively with prost-reflect — you do not compile schemas into the binary.

If the source fails to start (endpoint unreachable, schema missing), datavis logs the error and keeps running without live data rather than aborting.

Mapping messages to channels

Declare channels in config.toml with the proto field paths that carry the value and timestamp:

[channels."sensor.acceleration.x"]
topic      = "accel"
proto_path = "AccelBatch.samples.x"
ts_path    = "AccelBatch.samples.t_ns"
type       = "float"

• topic matches the ZMQ topic the message arrives on.
• proto_path / ts_path are dotted paths into the decoded message. Repeated fields (batches of samples) are expanded — one channel sample per element.

See Channels for every field.

The shared schema

All ZeroMQ channels share the one descriptor set compiled from --schema. When you record, that same schema is embedded into the MCAP file, so a recording is self-describing and replays without needing the original .proto on hand. See Recording to MCAP.

MQTT

The MQTT source subscribes to a broker and ingests Protobuf payloads. Unlike the ZeroMQ source, it discovers topics and builds their schema on the fly — you do not have to enumerate every topic ahead of time.

Enabling it

cargo run -- --mqtt-endpoint localhost:1883

--mqtt-endpoint <ADDR> takes host:port (or just host, defaulting to port 1883). Omit the flag to leave MQTT off.

Runtime topic discovery

As publishes arrive, datavis tracks the set of discovered topics. A topic that is not yet bound to a channel still appears in the sidebar's discovered list — drag it onto a panel to start plotting it. Discovered channels use the [defaults] buffer sizing.

You can also bind MQTT topics explicitly in config.toml, the same way as any other channel, if you want a fixed name, unit, colour, or EU scaling.

On-the-fly schema for recording

Each MQTT topic carries its own generated Protobuf schema, built from the payload the first time datavis sees the topic. The generated message has two fields — a nanosecond timestamp (t_ns) and a typed value. When you record, that per-topic schema is embedded into the MCAP file alongside the samples, so MQTT recordings are self-describing and replay exactly, without the live broker.

Timestamps are stamped at receipt (now_ns()), and also written into the message's t_ns field.

Backpressure

If the record queue is full, the sample is dropped (standard non-blocking try_send semantics) and the gap is accounted for — recording never blocks ingest.

WebSocket / Influx Line Protocol

The WebSocket source binds a WebSocket server and accepts InfluxDB line protocol text frames. It is the easiest way to push data in from a script or a device that already speaks line protocol, with no Protobuf schema required.

Enabling it

cargo run -- --ws-listen 127.0.0.1:8086

--ws-listen <ADDR> binds a WebSocket server on host:port that receives line protocol. Omit the flag to leave it off. Clients connect and stream text frames; conn_state transitions to LIVE once a client connects.

Line protocol

Each frame is one or more InfluxDB line-protocol measurements:

measurement,tag=value field1=1.23,field2=42i 1700000000000000000

Topic mapping

datavis maps a measurement + field key onto a channel topic, so a single measurement with several fields feeds several channels. Bind them in config.toml by topic, or drag discovered topics onto panels at runtime, just like MQTT.

Field value normalisation

Line protocol field values are normalised to datavis sample types:

• Floats → float
• Integers (the trailing-i form) → int
• Booleans (t/true/f/false) → bool
• Quoted strings → text

Timestamps

The optional trailing nanosecond timestamp is used when present; otherwise the sample is stamped at receipt.

When to use it

Reach for the WebSocket source when the producer already emits line protocol (Telegraf, an Influx client library, a quick script) and you would rather not define a Protobuf schema. For high-rate binary telemetry, prefer ZeroMQ or MQTT.

External Bridge Adapters

A bridge is a separate process that speaks a proprietary transport on one side and datavis's simple pipe protocol on the other. It lets an organisation feed data from a closed-source or vendor-specific system without touching any datavis source code.

Why a subprocess

datavis is licensed under GPL-3.0. A bridge adapter is a separate process communicating over a pipe — it does not link against any datavis library or include its source. That subprocess relationship is not a derivative work under the GPL, so your organisation may keep the adapter source closed.

Configuring a bridge

Bridges are declared in config.toml as an array of tables, because a bridge needs a command and arguments (not just an endpoint string) and several may run at once:

[[sources.bridge]]
name    = "vendor-x"                 # shown in the status bar / logs
command = "/opt/vendor-x/adapter"    # the org's proprietary executable
args    = ["--device", "/dev/tty0"]  # optional; defaults to empty

datavis spawns one subprocess per entry and owns its lifecycle — starting it at launch and tearing it down on exit.

Declaring bridge channels

Bridge channels carry a fixed, built-in payload schema, so they drop proto_path / ts_path and declare only the topic and sample type:

[channels."vx/accel_x"]
topic       = "accel"        # matches the Column.topic on the wire
sample_type = "f64"

Topic→channel resolution reuses the normal registry lookup. As always, topics must be unique across coexisting sources.

The wire contract

The adapter writes a one-time preamble, then a stream of frames, each carrying a Protobuf Batch of columns (topic + typed samples at possibly different rates). The full byte-level specification — preamble, framing, the Batch schema, and channel mapping rules — is the External Source Protocol guide shipped in the repo at docs/external-source-protocol.md.

Reference adapter

A minimal working adapter lives at examples/echo_bridge.rs. It writes the preamble once and emits a single Batch with three columns at different rates and types. Inspect its raw bytes:

cargo run --example echo_bridge | xxd | head

A real adapter swaps the hard-coded samples for data from your transport but keeps the same framing. The example is the complete wire contract in executable form.

Demo Source

The demo source synthesises channels so you can explore datavis without wiring up any real transport. It is the fastest way to see the app working.

Enabling it

cargo run -- --demo

• --demo — run with the built-in demo source and no live inputs.
• --demo-freq <HZ> — the sine frequency for the demo source. Default 1.0.

What it produces

The demo feeds a set of channels (sine waves at the configured frequency, plus a few state/discrete channels) into the store, exactly as a real source would. Because everything above the store reads through the same ChannelStore seam, panels, cursors, recording, and scripting all behave identically on demo data.

That makes the demo source useful beyond a first look:

• Try out panel layouts and the drag-and-drop workflow.
• Exercise scripting against known inputs.
• Record a short demo session and practise playback.

Note

When --demo is set, live transports are not started — the demo source stands in for them. Drop it (omit --demo) to go back to real sources.

Panel Types

datavis draws channels in a workspace of panels. Each panel type accepts certain sample types and is looked up through a registry keyed by the type string in the layout. Bind a channel of the wrong type and the panel renders an inline error — it never crashes.

The panels

Single-channel panels (gauge, numeric, status, state graph) take a channel key; multi-channel panels (waveform, spectrum) take a channels list; the xy_scatter panel takes separate x_channel and y_channel keys. See Panels & Layout.

Waveform

The workhorse time-series panel. Scrolls live, retains its configured window, and supports cursors plus min/max/mean/RMS measurements over a selection. It has a rich zoom gesture — see Zooming & Cursors. Layout keys include cursors and dots.

Spectrum (FFT)

Computes an FFT over the channel's current window with rustfft, with a configurable window function. It warns when the input timestamps are non-uniform (the transform assumes uniform sampling).

State graph

Renders bool / int channels as coloured horizontal bands over time — good for discrete states and modes. It interns text/int/bool snapshots itself, so it works on live and replayed data alike.

Status

A single current-value indicator for discrete channels (text, int, bool — not float, which has no discrete states). You map raw values to labelled, coloured states in the layout:

type    = "status"
channel = "motor.state"
label   = "Motor"
states  = [
  { match = "2", label = "FAULT",   color = "#d62728" },
  { match = "1", label = "RUNNING", color = "#2ca02c" },
  { match = "0", label = "IDLE",    color = "#7f7f7f" },
  { match = "true", label = "ON",   color = "#2ca02c" },
]

• match — the raw value key (required). Booleans match true / false.
• label — display text; defaults to match when omitted.
• color — #rrggbb (required). Entries with a missing/malformed colour are skipped on load.

An unmatched value falls back to the raw value. For continuous quantities use a gauge or numeric instead.

Log

A scrolling, filterable view of a text channel's lines, each timestamped. max_lines on the channel bounds retention.

Zooming & Cursors

Time-series panels — the waveform panel in particular — support a two-mode box zoom and cursor measurements. Zoom can also be linked across panels so they share one time window.

Box zoom on the waveform

Left-drag a box on the plot to zoom. The gesture has two modes:

• Plain drag → snap to the dominant axis. datavis compares the box width and height:
  • Wider than tall → X zoom (time). Horizontal scroll keeps running live on the other axis.
  • Taller than wide → Y zoom (value). The time axis is left untouched.
• Shift + drag → free box zoom. Both the X (time) and Y (value) axes are set from the box.

A minimum 5-pixel travel threshold applies to the axis being set, so a tiny drag does nothing. For a free zoom, an axis under the threshold is left unchanged — a mostly-horizontal Shift-drag still just zooms X.

While you drag, the panel shows a live preview of the box before it is applied on release.

Resetting zoom

Reset a panel's zoom to return it to the live, auto-scrolling window. There is also a global reset that clears zoom across all linked panels at once.

Linked time zoom

Turn on linked zoom from the toolbar to make participating time-series panels (waveform, state graph) share a single time window. Zoom or scrub one and the others follow, so a set of panels stays aligned on the same span — useful for correlating signals across panels. The linked window is applied on top of each panel's own effective window rule.

Cursors & measurements

The waveform panel (and other panels where it is meaningful) supports a cursor and a selection region. Over a selection, the panel computes and displays min / max / mean / RMS. Enable cursors with the panel's cursors layout key.

Recording to MCAP

datavis records a live session to an MCAP file — including the Protobuf schemas it discovered — and plays it back through the same panels. Because panels read through the ChannelStore seam, replay and live look identical: a panel never learns which one it is showing.

Starting and stopping

Recording is a manual toolbar Record / Stop control. It is available whenever any ingest source is active; with no source (e.g. a bare demo with recording unsupported) the toolbar shows "Recording unavailable (no ingest source)".

Every active source feeds one shared record queue, so a session mixing ZeroMQ, MQTT, and a bridge records them all together.

The file

A recording is written to recording_<secs>.mcap in the working directory, where <secs> is the session start time. The file is self-describing:

• ZeroMQ channels embed the shared descriptor set compiled from --schema.
• MQTT channels embed a per-topic generated schema (a t_ns timestamp field and a typed value), built on the fly as topics are discovered.

Because the schema travels inside the file, a recording replays without the original .proto or the live source — see Playback.

How writing works

The recorder runs on its own thread, draining the record queue:

• Each message is written to its MCAP channel with its log timestamp and a monotonic sequence number.
• The writer flushes about once a second.
• On stop, it drains the remaining queued messages and finalises the file (writer.finish()), which writes the summary and chunk index that make the file lazily loadable.

Backpressure and gaps

The record queue is bounded. If it fills — the recorder cannot keep up — new samples are dropped rather than blocking ingest, and the gap is accounted for. A write failure sets a record_failed flag surfaced in the status bar; the app keeps running.

Large sessions

A long session can produce a very large single file. datavis can auto-split a recording into size-bounded parts that replay as one stitched timeline — see Auto-Split by Size. And a recording larger than RAM still plays back, because the loader is lazy — see Larger-Than-RAM Playback.

Auto-Split by Size

When a live recording grows past a size limit, datavis finalises the current .mcap file and continues into a new one, so no single recording file grows without bound. The parts replay back as one stitched timeline automatically.

Configuration

Add a [recording] section to config.toml:

[recording]
# Auto-split live recordings once the .mcap on disk reaches this many MB.
# Parts are named recording_<secs>_000.mcap, _001, ... and replay stitched
# together. Omit the section or set 0 to keep a single file.
max_file_mb = 512

• max_file_mb — roll over when the on-disk .mcap reaches this many megabytes (post-compression). Omit the section, or set 0, to keep the single-file behaviour.

File naming

With a limit set, parts of one session share the session start timestamp and are numbered sequentially:

recording_1785959160_000.mcap
recording_1785959160_001.mcap
recording_1785959160_002.mcap

With no limit (section omitted or max_file_mb = 0) the filename stays the plain recording_<secs>.mcap — byte-for-byte the previous behaviour, unchanged.

How rollover works

• Split is by on-disk file size, measured after the recorder's flush — it matches the "keep files under X" intuition and accounts for compression.
• Rollover is approximate: it fires at the first size check past the limit, so a part may overshoot by up to roughly one chunk plus a moment of data. Acceptable for "keep files under ~X".
• On rollover the recorder finalises the current part (writing its summary and chunk index, so each part is self-contained), then opens the next part and continues the same message stream into it. No samples are lost or duplicated at the boundary.
• Each new part re-registers its channels, so every part is independently replayable and carries its own embedded schemas.

Replaying split recordings

Each part is a complete, self-describing MCAP file. To replay a split session, open all its parts together in the file dialog — the loader stitches them onto one timeline, order-independently. Nothing about playback changes.

Auto-expanding a session's sibling parts from a single pick is a documented future nicety; today you multi-select the parts.

What it does not change

The message write path, the one-second flush cadence, sequence numbering, and the record queue are all unchanged. Playback — stitching, the lazy loader, and every panel — is untouched. With no limit configured, behaviour is exactly as before this feature existed.

Playback

Playback opens a recorded MCAP file and replays it through the same panels you use live. It is an exclusive mode: live ingest pauses while you scrub a recording.

Opening a recording

Open a recording_*.mcap file from the toolbar / file dialog. To open a split session, select all of its parts together — the loader stitches them onto one timeline, order-independently.

Because recordings are self-describing, datavis reads the Protobuf schemas from the file itself. You do not need the original .proto, the --schema flag, or a live source to replay. Channels reconstructed from the file become droppable in the sidebar; leaving replay restores your live channel tree.

Controls

Replay exposes a playback clock over the recording's duration:

• Play / pause.
• Scrub anywhere on the timeline.
• Variable speed from 0.1× to 10×. The clock advances position at speed × wall-time and drives which part of the file is decoded.

Panels update from the store as the position moves, exactly as they do live.

Reading model

As the position moves, panels ask the playback store for a window of samples (snapshot(channel, window)) or the latest sample at or before a time (latest_at). The store answers from whichever chunk(s) cover the requested range. This on-demand decode is what makes very large recordings playable — see Larger-Than-RAM Playback.

Robustness

• A recording with a gap (from record-queue overflow) loads fine — the store returns whatever subset is present.
• A decode error inside a chunk is logged and that message skipped; the rest of the chunk still decodes.
• A topic re-registered with a conflicting type across parts is skipped rather than corrupting the timeline.

Leaving replay

Close replay to return to live mode. Your pre-replay channel tree and layout are restored.

Larger-Than-RAM Playback

datavis plays back recordings that do not fit in memory. Rather than loading a whole MCAP file into RAM, it memory-maps each file and decodes chunks on demand, so even the compressed bytes need not fit in RAM.

Why

A long or high-rate recording can easily exceed available memory. Eagerly retaining every decoded sample would make such files unopenable. The lazy loader removes that ceiling: memory use tracks what you are looking at, not the file size.

How it works

• Always lazy, one path. Every load memory-maps the file (via memmap2), builds an envelope of what is where, and decodes chunks only as panels request them. There is no separate small-file path — one implementation for all sizes.
• On-demand decode. When a panel asks for a window, the store decodes just the chunk(s) covering that range and serves the slice. Scrub elsewhere and different chunks are paged in; the OS pages the compressed bytes as needed.
• Single-level envelope. For wide zoom-outs the store serves a decimated min/max envelope rather than every sample. There is one envelope level, not a multi-resolution pyramid.
• Stitched across parts. Loading multiple files (a split session) builds one combined timeline over all of them.

Trade-offs and limits

These are deliberate, documented v1 limitations:

• Blocky mid-zoom fidelity. In the narrow band that is too wide for full-detail decode yet thin on envelope buckets, the trace can look blocky. A multi-level level-of-detail pyramid is future work.
• Text/log channels stay fully in RAM. Logs are low-volume, so this is cheap, but it is the one component that remains O(file) in memory. Spilling them to on-demand decode is future work.
• High-rate state graphs at full-file zoom show approximate bands. A high-rate int channel viewed at near-whole-file zoom uses the envelope, so its state bands are approximate. Low-rate state channels touch few chunks and stay on the exact detail path, so they are unaffected.

Error handling

• A missing or short summary falls back to a per-file message scan for bounds and a single whole-file span (that file loses chunk-level laziness but still loads).
• An mmap failure propagates with the file path in the error, just like a plain read failure would.

Future work

Multi-level LOD pyramids for smooth fidelity at every zoom, lazy (spill-to- decode) text channels, and prefetch of neighbouring chunks to hide scrub-time decode latency are all noted for later.

Python Scripting

datavis can derive new channels from existing ones by running Python scripts. Scripts are numba-compiled for speed and run live and during replay, feeding their outputs back into the store as ordinary channels you can drop on panels.

If numba (or the Python toolchain) is unavailable, scripting degrades gracefully — the engine disables itself with a warning and the rest of the app runs normally. Scripting is also a build-time feature (scripting, on by default); build with --no-default-features to drop the Python dependency entirely.

The scripts directory

Scripts live in a directory next to config.toml (default scripts/). The engine scans it for *.py files at startup — these are the available scripts. Availability is discovery; which scripts are enabled is separate and persisted in config (see Script Bindings).

The script contract

A script is a .py file that self-declares its bindings as module globals and implements a numba-compiled compute function. Because compute is compiled with numba it must be pure: tuples of numpy arrays in, (ts, vals) array pairs out. No dicts of objects, no .push(), no channel handles cross the @njit boundary — the engine handles all of that.

Element-wise transform

Magnitude of a co-sampled 3-axis vector:

import numpy as np
import numba

INPUTS  = ["accel.x", "accel.y", "accel.z"]
OUTPUTS = [{"name": "accel.magnitude", "type": "float", "unit": "m/s2"}]

@numba.njit
def compute(ts, vals):
    t = ts[0]                            # accel.x timestamps
    x, y, z = vals[0], vals[1], vals[2]
    mag = np.sqrt(x**2 + y**2 + z**2)
    return (t, mag)                      # one (ts, vals) pair per output

Multi-rate combine

The engine never resamples for you — do it yourself:

import numpy as np
import numba

INPUTS  = ["accel.x", "gps.speed"]
OUTPUTS = [{"name": "norm.speed", "type": "float", "unit": "m/s"}]

@numba.njit
def compute(ts, vals):
    tx, x = ts[0], vals[0]
    tg, g = ts[1], vals[1]
    g_on_x = np.interp(tx.astype(np.float64), tg.astype(np.float64), g)
    return (tx, g_on_x * x)

Window reduction

Reduce a window to a single sample:

import numpy as np
import numba

INPUTS  = ["motor.current"]
OUTPUTS = [{"name": "motor.current.rms", "type": "float", "unit": "A"}]

@numba.njit
def compute(ts, vals):
    t, v = ts[0], vals[0]
    rms = np.sqrt(np.mean(v**2))
    return (t[-1:], np.array([rms]))     # length-1 ts + vals -> one sample

Declaring the engine in config

[scripts]
dir = "scripts"          # scripts directory, relative to config.toml
window_s = 10.0          # sliding window of input handed to compute

• dir — scripts directory. Defaults to "scripts" when the section is absent.
• window_s — seconds of input history passed to each compute call.

The INPUTS / OUTPUTS in a script are its defaults. You override them per instance — reusing one script on different channels — in [[scripts.instances]].

Script Bindings

A script declares default inputs and outputs. A binding is one concrete use of that script — pointing its inputs at real channels and naming its outputs. Bindings let you reuse one script on several channel sets, each producing its own derived channels.

Instances in config

Each active binding is a [[scripts.instances]] entry:

[scripts]
dir = "scripts"
window_s = 10.0

[[scripts.instances]]
id = "ch0_squared"        # unique instance id
script = "sine_squared"   # script stem (file name without .py)
enabled = true            # inputs/outputs omitted → use the script's defaults

[[scripts.instances]]
id = "ch1_rms"
script = "sine_rms"
inputs  = ["load/ch0"]
outputs = [{ name = "scripts/rms", type = "float", unit = "" }]
enabled = true

[[scripts.instances]]
id = "diff"
script = "channel_diff"
inputs  = ["load/ch0", "load/ch7"]
outputs = [{ name = "scripts/diff", type = "float", unit = "" }]
enabled = true

• id — unique per instance; identifies it in the UI and config.
• script — the script file stem in the scripts dir.
• inputs — channel names to feed compute. Omit to use the script's own INPUTS. The count must match the script's arity.
• outputs — output channel definitions (name, type, unit). Omit to use the script's OUTPUTS.
• enabled — whether the binding runs.

Output-name templates

Output names may use placeholders that expand from the bound inputs, so one script reused on many inputs yields distinct output channels:

• {in0} — the full name of input 0.
• {in0.stem} — the last path segment of input 0.
• {in1}, {in1.stem}, … for further inputs.

A literal name passes through unchanged. An unknown placeholder is an error.

Resolution rules

• Arity must match. If the number of bound inputs (or outputs) does not match the script, the instance is marked Failed with a reason.
• Output names must be unique. Two instances declaring the same output channel collide; the second is Failed. Rebuilding or removing an instance releases the output names it owned, so re-applying does not self-collide.
• Unresolved inputs block Apply. In the panel editor, an input that does not resolve to a channel prevents applying that binding.

The script panel

The script panel is a full editor for bindings: pick a script, resolve its inputs against available channels (with a fuzzy filter that ranks matches), name the outputs, and enable it. Applying or removing a binding updates the store's derived channels live; changes persist to [[scripts.instances]].

Lifecycle & status

Each instance reports status — running, disabled, or Failed with a reason (arity mismatch, output collision, compile error). A failing script never takes down the app or other scripts; it is isolated and surfaced in the UI. If the whole engine cannot start (numba missing), it reports a single disabled reason and the app runs without derived channels.

Command-Line Options

datavis [OPTIONS]

Run datavis --help (or -h) to print this list.

Notes

• Channels and layout share config.toml; the layout section persists there.
• The status indicator reads LIVE if any configured source is receiving data.
• Sources can be combined: pass --endpoint, --mqtt-endpoint, and --ws-listen together to run all three. Bridges are configured in config.toml, not on the command line (see Bridge Adapters).
• If config.toml is absent in the working directory, datavis offers to save the built-in default, load another file, or run with defaults once.

Examples

# Fastest look — demo source
cargo run -- --demo

# Live ZeroMQ + Protobuf
cargo run -- --endpoint tcp://localhost:5555 --schema schema.proto

# Everything at once
cargo run -- \
  --endpoint tcp://localhost:5555 --schema schema.proto \
  --mqtt-endpoint localhost:1883 \
  --ws-listen 127.0.0.1:8086

config.toml Reference

One config.toml in the working directory holds every section below. Each concern parses the whole file and ignores sections it does not own, so sections are optional and order-independent. This page is the terse lookup; each section has a fuller chapter linked from its heading.

Top level

default_window_s = 10.0   # default visible time span for time-series panels

[channels."name"] — Channels

Bridge channels use topic + sample_type only (no proto_path / ts_path).

[defaults] — Defaults

[defaults]
max_rate  = 100000    # fallback Hz for channels omitting it
history_s = 5.0       # fallback seconds for channels omitting it

Only these two keys; per-channel values win. Omit for built-in defaults.

[screens.*] — Panels & Layout

GUI-managed. [screens.<name>] carries a list of [[screens.<name>.panels]] tables, each with a type, a frame = [x, y, w, h] (normalized canvas position), and either a channel or channels plus panel-specific keys.

[[sources.bridge]] — Bridge Adapters

[[sources.bridge]]
name    = "vendor-x"
command = "/opt/vendor-x/adapter"
args    = ["--device", "/dev/tty0"]   # optional

One subprocess per entry. Its channels are declared as normal [channels.*] with topic + sample_type.

[scripts] / [[scripts.instances]] — Scripting

[scripts]
dir = "scripts"          # scripts dir, relative to config.toml
window_s = 10.0          # input window seconds

[[scripts.instances]]
id = "ch1_rms"
script = "sine_rms"
inputs  = ["load/ch0"]
outputs = [{ name = "scripts/rms", type = "float", unit = "" }]
enabled = true

inputs / outputs are optional (fall back to the script's declared defaults).

[recording] — Auto-Split

[recording]
max_file_mb = 512   # auto-split at this on-disk size; omit or 0 = single file

Architecture

This chapter sketches how datavis is put together, for readers who want to extend it or understand why features behave the way they do.

The ChannelStore seam

The central abstraction is the ChannelStore trait. Everything above it — every panel — reads samples through this one interface and never learns whether the data is live or replayed. Two implementations sit behind it:

• LiveStore — live ring buffers plus text history, fed by ingest sources.
• PlaybackStore — reads from memory-mapped MCAP file(s), decoding chunks on demand.

This seam is why recording, replay, and larger-than-RAM playback "just work" through the same panels, and why the demo source is indistinguishable from a real one.

Module map

Sources

Each transport implements a common DataSource trait: build, spawn onto the store, and report a conn_state. The app collects sources into a list and treats them uniformly, so they compose. The MQTT-family sources share a scalar ingest helper.

Panels

Panels implement VizPanel (title, accepted types, config_ui, render, serialize) and are constructed through a PanelRegistry keyed by the layout's type string. A factory pattern is used because a trait object cannot call a Sized deserialize directly. Panels resolve channel names to ids through the ChannelRegistry, so they never parse names themselves.

Recording pipeline

Ingest sources push RecordMsg values onto a bounded queue. A dedicated recorder thread drains it into an mcap::Writer:

• ZMQ messages carry the shared descriptor set.
• MQTT messages carry a per-topic generated schema (DynamicProto).

The writer flushes ~once a second and, on stop or rollover, finalises the file with a summary and chunk index — the metadata the lazy playback loader relies on.

Playback pipeline

PlaybackStore::load_many(paths, registry) memory-maps each file, builds an envelope of chunk locations and bounds, and reads the embedded schemas so replay is self-describing. Reads (snapshot, latest_at) decode just the chunk(s) covering the requested window. Text channels are held fully in RAM; numeric channels use the on-demand envelope. See Larger-Than-RAM Playback.

Threading

Ingest sources, the recorder, and the script engine each run on their own threads and communicate with the UI through channels and shared state. The UI renders at its own cadence (~60 fps), decoupled from sample rates that can reach 100 kHz per channel.

Further reading

The per-feature design specs under docs/superpowers/specs/ and the implementation plans under docs/superpowers/plans/ carry the detailed rationale, decisions, and testing strategy behind each subsystem summarised here.