Skip to content

Telemetry

sal records OpenTelemetry traces around the steps that take time, and reports the memory and CPU the process used while they ran, so that a slow build or query can be read as a timeline rather than guessed at from the log.

Nothing is exported unless the environment asks for it. A plain sal build pays only for spans that go nowhere, except that the progress pane it shows in a terminal reads the spans as they start and end, which is how it follows the build; the spans are still not exported.

The exporters are configured with the standard OpenTelemetry environment variables, so any OTLP backend works. Set an endpoint and traces and metrics both go there:

Terminal window
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
sal build
Variable Effect
OTEL_EXPORTER_OTLP_ENDPOINT Base URL of an OTLP collector. /v1/traces and /v1/metrics are appended to it. Setting it turns both signals on.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT The full URL for one signal, used as written. Setting one turns on only that signal.
OTEL_EXPORTER_OTLP_PROTOCOL http/protobuf, the default, or grpc. OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL set it per signal.
OTEL_EXPORTER_OTLP_HEADERS Headers to send, as key=value,key2=value2, for a backend that wants a token.
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER otlp, console, or none. console pretty prints to stderr and needs no collector. Unset, each is otlp when an endpoint for it is configured and none otherwise.
OTEL_METRIC_EXPORT_INTERVAL How often metrics are exported, in milliseconds. The default is 10 seconds, and a final export happens as the process exits, so a short build still reports.
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES Rename the service, which is sal by default, or tag everything it sends.

To look at one build without running anything else:

Terminal window
OTEL_TRACES_EXPORTER=console sal build 2> trace.json

Only the steps that can take a while have spans. Everything else is covered by the span above it, so a trace reads as an outline of the invocation rather than a call graph.

sal build, sal run, and sal validate each produce one trace, named sal.build, sal.run, or sal.validate, carrying sal.files, how many source files were read. Under it:

Span What it covers Attributes
build.validate_sources Checking every term of every source file against its vocabulary, and pinning every declared prefix. This is where vocabularies are fetched. sal.documents
validate.document One source file’s terms. sal.file, sal.terms
validate.load_vocabulary Resolving one vocabulary, from the project’s pin or from its source. sal.vocabulary, sal.vocabulary.pinned, sal.vocabulary.bytes, sal.vocabulary.terms
salmodule.resolve Turning a salmodule:// reference into a container image. sal.module.cache says what was reused: pinned image when nothing was cloned or built, image when the clone’s commit already had an image, none when the image was built. sal.module, sal.module.commit, sal.module.image, sal.module.cache
salmodule.clone The git clone of the module repository. sal.module.clone_url
salmodule.build_image The docker build of the module’s Dockerfile. sal.module.image
salmodule.ontology Running the module’s salmodule ontology command for its vocabulary. sal.module
build.import_ontologies Fetching every owl:imports of the project ontology, with a build.import_ontology under it per import. Absent from a sal validate trace, which fetches none.
build.import_ontology Fetching one owl:imports of the project ontology. sal.import
build.materialize_modules Running every SAL module task the sources declare, with a salmodule.task under it per task. The counts are known up front, so a reader can tell how far along the tasks are. Absent from a sal build --without-sal-modules or sal validate trace. sal.module.tasks, sal.modules
salmodule.task One task instance from the module’s ontology through to its triples written to the build’s spool. sal.module.documents, the JSON-LD nodes read so far, and sal.triples.materialized are updated every 20,000 nodes while the task runs. sal.module, sal.module.task_class, sal.module.task, sal.module.documents, sal.triples.materialized, sal.module.files_copied
salmodule.run_task The module’s salmodule run command, from the container starting to the last file:/// copy finishing. sal.module, sal.module.output_bytes, sal.module.files_copied
build.construct Staging the build on a branch of the project’s table and running the project’s CONSTRUCT queries over it. The staging is an iceberg.stage under this span. Absent when the project has no CONSTRUCT query. sal.construct.queries, sal.triples.constructed
iceberg.stage Writing what the table does not hold yet to the sal-staging branch, for the CONSTRUCT queries to read. It has an iceberg.read_existing, iceberg.diff, and iceberg.commit of its own. sal.iceberg.warehouse, sal.iceberg.namespace
construct.query One CONSTRUCT query, from translating it to its triples written to .sal/constructed. sal.construct.query, sal.triples.constructed
sparql.construct Translating one CONSTRUCT query and streaming the rows of the SQL. sal.sparql.query
iceberg.write Writing the build to the Iceberg table. sal.iceberg.warehouse, sal.iceberg.namespace
iceberg.read_existing Writing the triple hashes the table holds, with its equality deletes applied, to a Parquet file for the diff to read. Absent for a table with no snapshot. sal.triples.existing
iceberg.diff Comparing the build’s spooled triples with the table’s in DuckDB, which decides what the snapshot adds and removes, and writing the triples it adds, with an iceberg.write_data_files under it. sal.triples.existing, sal.triples.added, sal.triples.removed, sal.triples.unchanged
iceberg.write_data_files Writing the new triples to Parquet data files, one partition at a time, each triple once. sal.iceberg.data_files, sal.triples.written
iceberg.commit Writing the equality deletes and committing the snapshot. sal.iceberg.data_files, sal.triples.added, sal.triples.removed

sal construct produces one trace, named sal.construct, with a construct.query and a sparql.construct span per query under it.

sal push, sal pull, and sal clone each produce one trace, named sal.push, sal.pull, or sal.clone, carrying sal.oci.repository. A sal.pull that pulled nothing, because the local data product already holds every snapshot the artifact does or is ahead of it, carries why as sal.pull.skipped, and has no oci.pull_layers span. The transfer spans update their attributes while they run, which is what the progress pane draws the transfer from; an exporter sees the final counts.

Span What it covers Attributes
oci.fetch_manifest Fetching the artifact’s manifest from the registry. In a sal push trace it is the check of which snapshots the remote already has, and ends cleanly whether or not the remote exists. sal.oci.reference
oci.push_layers Uploading every file of .sal/data as a layer, sal.oci.parallel at a time. sal.oci.bytes_done counts the bytes as the registry reads them, so it can exceed sal.oci.bytes when a body is sent again after an auth challenge. sal.oci.repository, sal.oci.parallel, sal.oci.files, sal.oci.bytes, sal.oci.files_done, sal.oci.bytes_done
oci.push_manifest Packing the manifest and tagging it latest and with the commit hash. sal.oci.repository, sal.oci.tags
oci.pull_layers Restoring every titled layer into the destination, sal.oci.parallel at a time, counting the bytes as they arrive. Also recorded by the oci:// imports sal build pulls. sal.oci.reference, sal.oci.destination, sal.oci.parallel, sal.oci.files, sal.oci.bytes, sal.oci.files_done, sal.oci.bytes_done
clone.repository The git clone of the source repository the artifact names. sal.clone.source
clone.checkout The git checkout of the commit the artifact was built from. sal.clone.commit
clone.init The sal init of the cloned repository, or, in a sal pull trace, the initialization of the empty directory the data product is pulled into; absent from a pull inside a project. sal.clone.directory

Every request sal makes to the docker daemon while resolving or running a module is a span of its own under the step that made it, named after what the request does rather than after its path: docker.image.build, docker.image.inspect, docker.container.create, docker.container.start, docker.container.logs, docker.container.wait, docker.container.kill, docker.container.remove, docker.container.copy, and docker.ping. The HTTP attributes the docker client records, such as url.full, still name the container.

A span whose step failed carries the error as its status and as an exception event, so a failed build shows where it stopped.

sal serve produces one trace per request, named by the method and path, such as POST /sparql or GET /v7847372623449923428/sparql, and honours the W3C trace context headers a client sends, so a request made from an instrumented application joins that application’s trace. Requests for the UI’s own static assets are left out. Under a request:

Span What it covers Attributes
sparql.query Translating a SPARQL query and running the SQL. sal.sparql.query
duckdb.query One DuckDB statement, which is also what /api/sql, /api/stats, and the geometry endpoints run. sal.duckdb.demoted is set when sal serve moved the statement to its second database for running past --slow-query-after. db.system.name, db.query.text, db.response.returned_rows, sal.duckdb.demoted
duckdb.explain Planning one DuckDB statement without running it, which is what /api/sql/explain does. db.system.name, db.query.text
duckdb.open Opening DuckDB and loading the Iceberg extension, once per process on the first query.
duckdb.load_spatial Loading the spatial extension the first time a query needs it.

Query text is cut to 4096 bytes. sal query records the same sparql.query and duckdb.query spans, one trace per statement.

Metrics are collected at OTEL_METRIC_EXPORT_INTERVAL and once more as the process exits. sal defines no metrics of its own; it reports the OpenTelemetry Go runtime and host instrumentation, which cover:

  • Memory: go.memory.used, go.memory.limit, go.memory.allocated, go.memory.allocations, and go.memory.gc.goal for the Go heap, and system.memory.usage and system.memory.utilization for the machine. A build that parses a large graph shows up here before it shows up anywhere else.
  • CPU: process.cpu.time for the sal process and system.cpu.time for the machine, both split by state, so a step that waits on Docker or the network can be told from one that computes.
  • Scheduling: go.goroutine.count, go.processor.limit, go.schedule.duration, and go.config.gogc.
  • Requests: http.server.request.duration from sal serve, a histogram per method and status.
Developed byCenter for Geospatial SolutionsCenter for Geospatial Solutions