Telemetry
sal records OpenTelemetry traces around the steps that take time, and reports the memory and CPU the process used while they ran, so that a slow build or query can be read as a timeline rather than guessed at from the log.
Nothing is exported unless the environment asks for it.
A plain sal build pays only for spans that go nowhere, except that the progress pane it shows in a terminal reads the spans as they start and end, which is how it follows the build; the spans are still not exported.
Turning it on
Section titled “Turning it on”The exporters are configured with the standard OpenTelemetry environment variables, so any OTLP backend works. Set an endpoint and traces and metrics both go there:
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318sal build| Variable | Effect |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT |
Base URL of an OTLP collector. /v1/traces and /v1/metrics are appended to it. Setting it turns both signals on. |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT |
The full URL for one signal, used as written. Setting one turns on only that signal. |
OTEL_EXPORTER_OTLP_PROTOCOL |
http/protobuf, the default, or grpc. OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL set it per signal. |
OTEL_EXPORTER_OTLP_HEADERS |
Headers to send, as key=value,key2=value2, for a backend that wants a token. |
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER |
otlp, console, or none. console pretty prints to stderr and needs no collector. Unset, each is otlp when an endpoint for it is configured and none otherwise. |
OTEL_METRIC_EXPORT_INTERVAL |
How often metrics are exported, in milliseconds. The default is 10 seconds, and a final export happens as the process exits, so a short build still reports. |
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES |
Rename the service, which is sal by default, or tag everything it sends. |
To look at one build without running anything else:
OTEL_TRACES_EXPORTER=console sal build 2> trace.jsonWhat is traced
Section titled “What is traced”Only the steps that can take a while have spans. Everything else is covered by the span above it, so a trace reads as an outline of the invocation rather than a call graph.
sal build, sal run, and sal validate each produce one trace, named sal.build, sal.run, or sal.validate, carrying sal.files, how many source files were read.
Under it:
| Span | What it covers | Attributes |
|---|---|---|
build.validate_sources |
Checking every term of every source file against its vocabulary, and pinning every declared prefix. This is where vocabularies are fetched. | sal.documents |
validate.document |
One source file’s terms. | sal.file, sal.terms |
validate.load_vocabulary |
Resolving one vocabulary, from the project’s pin or from its source. | sal.vocabulary, sal.vocabulary.pinned, sal.vocabulary.bytes, sal.vocabulary.terms |
salmodule.resolve |
Turning a salmodule:// reference into a container image. sal.module.cache says what was reused: pinned image when nothing was cloned or built, image when the clone’s commit already had an image, none when the image was built. |
sal.module, sal.module.commit, sal.module.image, sal.module.cache |
salmodule.clone |
The git clone of the module repository. |
sal.module.clone_url |
salmodule.build_image |
The docker build of the module’s Dockerfile. |
sal.module.image |
salmodule.ontology |
Running the module’s salmodule ontology command for its vocabulary. |
sal.module |
build.import_ontologies |
Fetching every owl:imports of the project ontology, with a build.import_ontology under it per import. Absent from a sal validate trace, which fetches none. |
|
build.import_ontology |
Fetching one owl:imports of the project ontology. |
sal.import |
build.materialize_modules |
Running every SAL module task the sources declare, with a salmodule.task under it per task. The counts are known up front, so a reader can tell how far along the tasks are. Absent from a sal build --without-sal-modules or sal validate trace. |
sal.module.tasks, sal.modules |
salmodule.task |
One task instance from the module’s ontology through to its triples written to the build’s spool. sal.module.documents, the JSON-LD nodes read so far, and sal.triples.materialized are updated every 20,000 nodes while the task runs. |
sal.module, sal.module.task_class, sal.module.task, sal.module.documents, sal.triples.materialized, sal.module.files_copied |
salmodule.run_task |
The module’s salmodule run command, from the container starting to the last file:/// copy finishing. |
sal.module, sal.module.output_bytes, sal.module.files_copied |
build.construct |
Staging the build on a branch of the project’s table and running the project’s CONSTRUCT queries over it. The staging is an iceberg.stage under this span. Absent when the project has no CONSTRUCT query. |
sal.construct.queries, sal.triples.constructed |
iceberg.stage |
Writing what the table does not hold yet to the sal-staging branch, for the CONSTRUCT queries to read. It has an iceberg.read_existing, iceberg.diff, and iceberg.commit of its own. |
sal.iceberg.warehouse, sal.iceberg.namespace |
construct.query |
One CONSTRUCT query, from translating it to its triples written to .sal/constructed. |
sal.construct.query, sal.triples.constructed |
sparql.construct |
Translating one CONSTRUCT query and streaming the rows of the SQL. |
sal.sparql.query |
iceberg.write |
Writing the build to the Iceberg table. | sal.iceberg.warehouse, sal.iceberg.namespace |
iceberg.read_existing |
Writing the triple hashes the table holds, with its equality deletes applied, to a Parquet file for the diff to read. Absent for a table with no snapshot. | sal.triples.existing |
iceberg.diff |
Comparing the build’s spooled triples with the table’s in DuckDB, which decides what the snapshot adds and removes, and writing the triples it adds, with an iceberg.write_data_files under it. |
sal.triples.existing, sal.triples.added, sal.triples.removed, sal.triples.unchanged |
iceberg.write_data_files |
Writing the new triples to Parquet data files, one partition at a time, each triple once. | sal.iceberg.data_files, sal.triples.written |
iceberg.commit |
Writing the equality deletes and committing the snapshot. | sal.iceberg.data_files, sal.triples.added, sal.triples.removed |
sal construct produces one trace, named sal.construct, with a construct.query and a sparql.construct span per query under it.
sal push, sal pull, and sal clone each produce one trace, named sal.push, sal.pull, or sal.clone, carrying sal.oci.repository.
A sal.pull that pulled nothing, because the local data product already holds every snapshot the artifact does or is ahead of it, carries why as sal.pull.skipped, and has no oci.pull_layers span.
The transfer spans update their attributes while they run, which is what the progress pane draws the transfer from; an exporter sees the final counts.
| Span | What it covers | Attributes |
|---|---|---|
oci.fetch_manifest |
Fetching the artifact’s manifest from the registry. In a sal push trace it is the check of which snapshots the remote already has, and ends cleanly whether or not the remote exists. |
sal.oci.reference |
oci.push_layers |
Uploading every file of .sal/data as a layer, sal.oci.parallel at a time. sal.oci.bytes_done counts the bytes as the registry reads them, so it can exceed sal.oci.bytes when a body is sent again after an auth challenge. |
sal.oci.repository, sal.oci.parallel, sal.oci.files, sal.oci.bytes, sal.oci.files_done, sal.oci.bytes_done |
oci.push_manifest |
Packing the manifest and tagging it latest and with the commit hash. |
sal.oci.repository, sal.oci.tags |
oci.pull_layers |
Restoring every titled layer into the destination, sal.oci.parallel at a time, counting the bytes as they arrive. Also recorded by the oci:// imports sal build pulls. |
sal.oci.reference, sal.oci.destination, sal.oci.parallel, sal.oci.files, sal.oci.bytes, sal.oci.files_done, sal.oci.bytes_done |
clone.repository |
The git clone of the source repository the artifact names. |
sal.clone.source |
clone.checkout |
The git checkout of the commit the artifact was built from. |
sal.clone.commit |
clone.init |
The sal init of the cloned repository, or, in a sal pull trace, the initialization of the empty directory the data product is pulled into; absent from a pull inside a project. |
sal.clone.directory |
Every request sal makes to the docker daemon while resolving or running a module is a span of its own under the step that made it, named after what the request does rather than after its path: docker.image.build, docker.image.inspect, docker.container.create, docker.container.start, docker.container.logs, docker.container.wait, docker.container.kill, docker.container.remove, docker.container.copy, and docker.ping. The HTTP attributes the docker client records, such as url.full, still name the container.
A span whose step failed carries the error as its status and as an exception event, so a failed build shows where it stopped.
sal serve produces one trace per request, named by the method and path, such as POST /sparql or GET /v7847372623449923428/sparql, and honours the W3C trace context headers a client sends, so a request made from an instrumented application joins that application’s trace.
Requests for the UI’s own static assets are left out.
Under a request:
| Span | What it covers | Attributes |
|---|---|---|
sparql.query |
Translating a SPARQL query and running the SQL. | sal.sparql.query |
duckdb.query |
One DuckDB statement, which is also what /api/sql, /api/stats, and the geometry endpoints run. sal.duckdb.demoted is set when sal serve moved the statement to its second database for running past --slow-query-after. |
db.system.name, db.query.text, db.response.returned_rows, sal.duckdb.demoted |
duckdb.explain |
Planning one DuckDB statement without running it, which is what /api/sql/explain does. |
db.system.name, db.query.text |
duckdb.open |
Opening DuckDB and loading the Iceberg extension, once per process on the first query. | |
duckdb.load_spatial |
Loading the spatial extension the first time a query needs it. |
Query text is cut to 4096 bytes.
sal query records the same sparql.query and duckdb.query spans, one trace per statement.
What is measured
Section titled “What is measured”Metrics are collected at OTEL_METRIC_EXPORT_INTERVAL and once more as the process exits.
sal defines no metrics of its own; it reports the OpenTelemetry Go runtime and host instrumentation, which cover:
- Memory:
go.memory.used,go.memory.limit,go.memory.allocated,go.memory.allocations, andgo.memory.gc.goalfor the Go heap, andsystem.memory.usageandsystem.memory.utilizationfor the machine. A build that parses a large graph shows up here before it shows up anywhere else. - CPU:
process.cpu.timefor thesalprocess andsystem.cpu.timefor the machine, both split by state, so a step that waits on Docker or the network can be told from one that computes. - Scheduling:
go.goroutine.count,go.processor.limit,go.schedule.duration, andgo.config.gogc. - Requests:
http.server.request.durationfromsal serve, a histogram per method and status.

