Skip to content

Publishing Data

SAL can publish the same built data product in two ways:

  • sal push packages .sal/data as an OCI artifact.
  • sal upload deploys .sal/data to an object store bucket.

Build locally first with sal build.

Use sal push when you want a portable artifact that records the source Git repository and commit hash:

Terminal window
sal push ghcr.io/my-org/my-data-product --username "$OCI_USERNAME" --password "$OCI_PASSWORD"

The pushed artifact is tagged as latest and with the current Git commit hash. SAL adds OCI annotations that connect the artifact back to the source repository and commit, and its description ends with the --message of the latest build, so the registry says what the data product last changed.

Consumers can use sal clone to clone the source repository, check out the pinned commit, initialize SAL, and restore the data product into .sal/data.

Use sal upload when readers should query the Iceberg table directly from an object store:

Terminal window
sal upload --bucket gs://my-bucket/sal/triples

Supported object-store URL forms include gs://, s3://, azblob://, and Google Cloud Storage HTTPS URLs such as:

Terminal window
sal upload --bucket https://storage.googleapis.com/my-bucket/sal/triples

During upload, SAL stages a temporary copy of .sal/data, rewrites Iceberg metadata paths to the destination, and uploads the staged files. The local .sal/data directory is left unchanged.

After uploading to object storage, query the table with DuckDB’s Iceberg extension:

SET enable_object_cache = false;
SELECT subject, predicate, object
FROM iceberg_scan('gs://my-bucket/sal/triples')
LIMIT 5;

For more detail about where SAL writes generated files, see Data Layout.

sal terraform writes a Terraform configuration that serves the data product from Cloud Run, reading it from a Cloud Storage bucket, and sal init writes the same files with --with-iac. The project’s Makefile runs it:

Terminal window
sal terraform --project my-gcp-project # writes terraform/; or sal init --with-iac
make apply # terraform init && terraform apply
sal build --message "Add the station data" data/
make upload # sal upload --bucket gs://<bucket>/
terraform -chdir=terraform output service_url

What it creates:

  • A bucket in the chosen region, with uniform access and public access prevention, holding the copy of .sal/data that sal upload --bucket gs://<bucket>/ writes: the Iceberg table under the project’s name, the pinned vocabulary documents under blobs/, and the STAC catalog under stac/.
  • A service account with roles/storage.objectViewer on that bucket and nothing else.
  • A Cloud Run service running the sal image with the bucket mounted read-only at /mnt/sal through Cloud Storage FUSE, started as sal serve --data /mnt/sal. DuckDB reads the Iceberg table as local files through the mount, so the service account’s access is the only credential involved and no key is written anywhere. The service is allUsers-invokable by default, answering SPARQL at /sparql, the catalog at /stac/catalog.json, and the vocabulary documents at /blobs/.

The bucket and the service are created in one region, so a query never crosses one; the service runs in the second generation environment the FUSE mount needs, with CPU allocated only while a request is handled and a startup boost for DuckDB’s open; and the mount caches file metadata for five minutes, since Iceberg files never change in place and sal upload writes new names, so a running instance sees a fresh upload within that time and a new one at once. The defaults are deliberately small (one CPU, 2 GiB, no warm instance, at most two, twenty concurrent requests) and all of them are variables in terraform/variables.tf; min_instances = 1 removes the cold start at a steady cost, and with_ui = true serves the browser UI, which is off by default because its SQL endpoint runs arbitrary DuckDB statements against whatever the service can reach. make destroy removes the service and the bucket; the bucket must be emptied first unless bucket_force_destroy = true, so a destroy never deletes a data product by accident.

terraform/terraform.tfvars holds the project ID and is meant to be committed; the state file is gitignored and belongs in a remote backend once more than one person applies the configuration.

Developed byCenter for Geospatial SolutionsCenter for Geospatial Solutions