Data Layout
SAL separates source files from generated data.
Your Git repository contains RDF source files and project code; .sal/data contains generated output.
Local layout
Section titled “Local layout”After sal init, a project has:
.sal/ config.jsonld data/Makefilesparql/terraform/ with --with-iac, or after sal terraformMakefile documents the commands a project runs, each target one sal command, plus the terraform targets that deploy it; see sal init. .claude/skills/ and .agents/skills/ hold the agent skill when sal init was told to install it.
sparql/ is for the project’s SPARQL queries. It is source rather than generated data, so it sits in the project root and is not gitignored.
Every CONSTRUCT query in it, a file ending in .rq or .sparql, is run by sal build, which commits the triples it constructs with the rest of the build.
.sal/constructed holds what each of those queries constructed, one Turtle file per query named after it, written by sal build and by sal construct. It is gitignored, wiped before every run, and removed by sal clean --wipe. It is a copy for reading: the constructed triples themselves are in the table.
After sal build, .sal/data contains the built data product.
For the main triples table, SAL writes an Apache Iceberg table under the project namespace:
.sal/ config.jsonld data/ blobs/ <sha256-of-a-vocabulary-document> <sha256-of-a-file-a-module-task-produced> prov.jsonld <sha256-of-a-directory-a-module-task-produced>/ stac/ catalog.json triples/ collection.json <git-project-name>/ triples/ data/ metadata/The metadata directory contains Iceberg table metadata, manifest lists, manifests, and version-hint.text.
The data directory contains the table’s data files.
While a build runs, .sal/data/.scratch holds what it is about to commit, as Parquet, and the files DuckDB spills
to while it compares the build with the table; it can grow to roughly the size of the module output as Parquet, and
it is removed when the build ends. One a build that was killed left behind is cleared by the next build, never
uploaded, and removed by sal clean --wipe.
The files named by a hash under .sal/data/blobs are the vocabulary documents the project pins, described in
Pinned vocabularies below, and the files SAL module tasks produce, which
sal build copies out of the module’s container under the SHA-256 digest of
their contents. sal serve serves both over HTTP at /blobs/{sha256}.
A directory a task produces, a Zarr store or STAC catalog for instance, is copied with its files verbatim under
{digest}/, the SHA-256 of its contents, so that it can be read in place from disk, from /blobs/{digest}/, or
from a bucket the data product is uploaded to, and /blobs/{digest} serves it whole as a zip archive. Everything
under .sal/data/blobs other than prov.jsonld is named by a hash. prov.jsonld is the log of every file and directory tasks have copied there,
each under its digest with the module that produced it; a run adds to it and never removes from it. See
Files a task produces.
.sal/data/stac is the STAC catalog the build writes last, describing the table it just committed; see STAC catalog below. sal serve serves it at /stac/catalog.json.
RDF triples
Section titled “RDF triples”SAL builds RDF into rows representing triples, with the object split across one column per kind of term so that each can be compared and read natively:
subjectpredicateobject_string: a string or any other literal without a typed column, or a blank node with its_:prefixobject_iri: an IRI objectobject_geometry: a GeoSPARQLwktLiteral, stored as an Iceberggeometryobject_byte: anxsd:byteliteral, stored as anINTEGER(Iceberg has no 8-bit type)object_integer: an integer-valued XSD literal (xsd:integer,xsd:int,xsd:long, …), stored as aBIGINTobject_float: anxsd:float,xsd:double, orxsd:decimalliteral, stored as aDOUBLEobject_time: anxsd:dateTimeliteral with an explicit timezone, stored as aTIMESTAMPnormalized to UTCobject_type: the datatype IRI the literal object was written with;NULLfor an IRI or blank node objectobject_language: the language tag of anrdf:langStringliteral object, such asenfor"hello"@en;NULLfor every other objecttriple_hash: the SHA-256 identifying the triple, stored as its 32 raw bytes (an Icebergfixed[32], a DuckDBBLOB);hex(triple_hash)renders it as hex
Every row is a statement the project makes: what its source files assert, what it imports with owl:imports, what its modules produced, and the provenance of its pins.
A vocabulary the project only validates against contributes no rows, so a SPARQL query reasons over exactly what the project asserted or imported; see Reasoning over the graph.
Exactly one of the object value columns is set on each row, chosen by the literal’s declared datatype rather than by what its text happens to parse as. A typed literal whose lexical form does not fit its column — an out-of-range xsd:byte, a zoneless or sub-microsecond xsd:dateTime — is stored in object_string, and any datatype without a typed column (booleans, dates, custom types) lands there too. object_type keeps the exact datatype in every case, and object_language keeps the language tag of a language-tagged string, so exporting the table back to RDF (sal export) reproduces the original typed or tagged literal even when the value was stored as a string. The geometry column is what makes the table an Iceberg format version 3 table.
The hash identifies a triple from its subject, predicate, object, and the object’s datatype and language tag, so "chat"@en and "chat"@fr are two rows.
SAL uses this identity to avoid writing duplicate triples when the table already contains the same data, and removes a triple a later build no longer states with an equality delete on it.
It is written without dictionary encoding, since no two rows share a value.
A table built before the hash was stored as bytes held it as a 64-character hex string; sal build refuses such a table and asks for sal clean --wipe and a rebuild, since Iceberg cannot change a column’s type from string to fixed.
Table layout
Section titled “Table layout”How the triples table is partitioned, sorted, encoded, and indexed, which is what decides how much of it a query can skip.
The settings live in build/load/iceberg_properties.go (NewIcebergTableFromCfg, columnWriteProperties, applyWriteProperties), and the row order in spoolDiff.added in build/load/diff.go.
| Column | Partition | Sort | Bloom filter | Dictionary | Min/max statistics |
|---|---|---|---|---|---|
predicate |
truncate[20] |
1st key, ASC | — | forced on | yes |
subject |
— | 2nd key, ASC | 128 KiB | on (iceberg-go default) | yes |
triple_hash |
— | 3rd key in the rows only, for de-duplication; not declared | — | forced off | yes |
object_iri |
— | — | 128 KiB | on (iceberg-go default) | yes |
object_string |
— | — | — | on (iceberg-go default) | yes |
object_type |
— | — | — | on (iceberg-go default) | yes |
object_geometry |
— | — | — | forced off | Parquet: bounding box only; Iceberg: bounding box |
object_float |
— | — | — | forced off | yes |
object_integer |
— | — | — | forced off | yes |
object_byte |
— | — | — | forced off | yes |
object_time |
— | — | — | forced off | yes |
object_language |
— | — | — | on (iceberg-go default) | yes |
Partition
Section titled “Partition”There is one partition field, predicate_partition: the first 20 characters of the predicate (truncate[20]).
- Declared: new tables declare (predicate ASC, subject ASC) as their Iceberg sort order. iceberg-go also sorts each batch it writes by that order.
- Physical: the build’s diff (
spoolDiff.added) writes rows sorted bypredicate, subject, triple_hash.
Within a predicate, every row group covers its own range of subjects. A pattern with a constant subject, or a join on a few subjects, therefore reads about one row group per predicate rather than nearly all of them. This only shows once a predicate spans several row groups of 122,880 rows.
Why triple_hash is the third key
Section titled “Why triple_hash is the third key”-
De-duplication: the spool a build writes can hold the same triple more than once, for example when a module writes a node twice.
distinctRowsremoves the copies as the sorted rows stream into the Iceberg writer, by dropping a row whose hash equals the hash of the row just before it. That only works if every copy of a triple sits next to its first copy. Copies share a predicate and a subject, but so do different triples with the same predicate and subject, so sorting by those two alone can separate them:s p "a" kepts p "b" kepts p "a" a copy of the first row, but not next to it, so it would be written againWith the hash as the last key, both copies of
s p "a"are adjacent and the second is dropped. Sorting on the hash is cheaper than de-duplicating with aDISTINCTover whole rows, which DuckDB cannot spill to disk. -
Deterministic order: the hash also fixes the order of rows that share a predicate and subject, so a rebuild of the same data writes the same rows in the same order.
Why it is not part of the declared order
Section titled “Why it is not part of the declared order”- Nothing that reads the table uses it. Only sal’s own diff needs the hash in the sort. DuckDB plans queries the same way whether or not a sort order is declared.
- It would cost something on every build. iceberg-go sorts each batch it writes by the declared order, so declaring the hash would add a third key to every one of those sorts for no benefit.
Bloom filters
Section titled “Bloom filters”subjectandobject_irionly, 128 KiB per column per row group (write.parquet.bloom-filter-max-bytes), which is about a 2% false positive rate for a full row group of 122,880 distinct IRIs.- What reads them: DuckDB probes the filter of each row group whose bounds could not rule a value out, for a constant equality filter or for the handful of keys a join pushes into the scan, and skips the row group when the IRI is not there. The object columns are not sorted, so for
object_irithe filter is what rules row groups out. - Fixed size: iceberg-go gives arrow-go no estimate of distinct values, so every filter is written at that size whatever the row group holds.
Dictionary encoding
Section titled “Dictionary encoding”- Forced on:
predicate(write.parquet.dict-encoding-enabled.column.predicate). - Forced off:
triple_hashandobject_geometry, whose values are unique or nearly so, and the numeric and time columns (object_byte,object_integer,object_float,object_time), which are almost always null and otherwise hold measured values that rarely repeat. - On, as iceberg-go’s default (
subject,object_iri,object_string,object_type,object_language):parquet.enable.dictionarydefaults totrue. iceberg-go also enables a fallback on every column: the writer switches a column chunk to plain encoding once its dictionary would outgrow the 2 MiB dictionary page (write.parquet.dict-size-bytes) or plain encoding would be smaller.
Min/max statistics
Section titled “Min/max statistics”There are two levels, both on data files:
- Iceberg file bounds (in the manifests): full metrics on every column (
write.metadata.metrics.default=full, pluswrite.metadata.metrics.column.object_geometry=full). That means untruncated lower and upper bounds, with null and value counts. - Parquet row-group statistics (in each file’s footer): min, max, and null count for every column except
object_geometry.
object_geometry is the exception:
- Parquet: no min/max, since GEOMETRY has no defined sort order. Instead each row group carries native geospatial statistics (bounding box and geometry types), written by arrow-go since apache/arrow-go#1504.
- Missing null count: there is no
Statisticsblock for the column at all, not even a null count, until apache/arrow-go#1533. DuckDB v2.0.0 ignores the bounding box without one. - Iceberg: the manifest bounds are a bounding box per file, which iceberg-go computes itself.
Outside the data files
Section titled “Outside the data files”- The build’s spool (temporary Parquet in
.sal/data/.scratch): dictionaries off forsubject,object_string,object_iri,object_geometry, andtriple_hash, and no statistics. - Equality delete files: each row names a deleted row by
triple_hash, the table’s identifier field (field 12). They are partitioned by predicate like the data, and their hash column has no dictionary.
Validation and vocabularies
Section titled “Validation and vocabularies”Before writing output, SAL validates RDF source files. Source data can be Turtle or JSON-LD.
Validation checks that terms used through prefixes are defined by the provided vocabularies.
For example, a misspelled term such as schema:nameee should fail validation rather than becoming a built triple.
Use sal validate to run these checks without writing a new build.
The project config file
Section titled “The project config file”.sal/config.jsonld is a single JSON-LD document with one @graph that condenses everything a project records about itself and the vocabularies it resolves against: the project’s own node (what sal init and sal import write) and the pinned vocabulary versions (what sal build writes). The two used to be separate files, .sal/ontology.jsonld and .sal/ns-prefix-versions.jsonld; they are now nodes in the same @graph, so a project has a single file describing everything it resolves and materializes against.
The first node in the graph is the project itself, "@id": ".", resolved against the base IRI the Git remote gives. sal init writes it as a schema:Project titled after the Git repository, the last segment of the remote URL, and carrying the description it asked for:
{ "@id": ".", "@type": "schema:Project", "dcterms:title": "my-project", "dcterms:description": "A demo project for sal; a small dataset of stream gauges and the agency that runs them"}The first sal import types the same node owl:Ontology and records the imports on it, keeping the title and description:
{ "@id": ".", "@type": ["schema:Project", "owl:Ontology"], "dcterms:title": "my-project", "dcterms:description": "A demo project for sal; a small dataset of stream gauges and the agency that runs them", "owl:imports": [{ "@id": "http://www.w3.org/2002/07/owl#" }], "rdfs:comment": "Represents the ontology for the my-project project overall and all the vocabularies that are directly materialized into the graph, not just referenced."}dcterms:description is what sal init asked for or was given with --description, and sal serve --with-ui shows it under the data product’s name. dcterms:title is the Git project name and sal import --title replaces it; rdfs:comment is always generated. Ontologies listed with owl:imports are merged at the version pinned for them as the project’s own statements. Pinning a vocabulary does not import it: a vocabulary the project validates against but does not import has no statements in the triples table, only the provenance node recording which version was pinned. The node’s own statements are validated and committed like a source file, which is why a build pins schema.org and dcterms for it.
Pinned vocabularies
Section titled “Pinned vocabularies”Vocabularies are resolved per project rather than from a shared cache on the machine.
sal build records the exact version of every vocabulary the project resolves against as the rest of .sal/config.jsonld’s @graph, one node per vocabulary, distinguished from the project node by carrying owl:versionIRI:
{ "@context": { "owl": "http://www.w3.org/2002/07/owl#", "dcterms": "http://purl.org/dc/terms/", "xsd": "http://www.w3.org/2001/XMLSchema#", "rdfs": "http://www.w3.org/2000/01/rdf-schema#", "schema": "https://schema.org/" }, "@graph": [ { "@id": ".", "@type": ["schema:Project", "owl:Ontology"], "dcterms:title": "my-project", "dcterms:description": "A demo project for sal; a small dataset of stream gauges and the agency that runs them", "rdfs:comment": "Represents the ontology for the my-project project overall and all the vocabularies that are directly materialized into the graph, not just referenced." }, { "@id": "https://schema.org/", "@type": "owl:Ontology", "rdfs:label": "https://schema.org/", "owl:versionIRI": { "@id": "urn:sha256:c3a1…" }, "dcterms:format": "application/ld+json", "dcterms:modified": { "@value": "2026-08-11T17:21:12Z", "@type": "xsd:dateTime" }, "rdfs:comment": "Represents the cached version of the https://schema.org/ vocabularies as of 2026-08-11T17:21:12Z" } ]}The document behind each pin is stored under .sal/data/blobs named by that same SHA-256.
Each pinned node carries the vocabulary’s namespace IRI as its rdfs:label, the human readable name it is known by, and the same node is committed into the triples table as provenance on every build.
config.jsonld belongs in Git; the documents it names do not, since sal init adds .sal/data to .gitignore and they travel with the data product instead.
On each build, SAL:
- Collects every prefix the project’s source files declare, building any SAL module it references so the module’s own ontology counts as one of them.
- Resolves each prefix that is not already pinned, hashes the document it received, writes it under
.sal/data/blobs, and records the mapping. - Resolves every prefix that is already pinned from the stored document rather than from the network.
Validation therefore checks terms against the exact version of a vocabulary the project pinned, not against whatever its publisher is serving today. A prefix that is declared but never used is pinned too, so adding a term from it later does not silently change which version the project validates against. A pin whose document is missing locally — as in a fresh clone that has not pulled the data product — is fetched again rather than failing the build.
Use --no-cache on sal build to resolve every prefix from its source again and re-pin it, which is how a project moves to newer vocabulary versions.
Querying pinned vocabularies
Section titled “Querying pinned vocabularies”Every pinned vocabulary node sal build writes to .sal/config.jsonld is also written into the triples table itself, as an owl:Ontology node carrying the same owl:versionIRI, dcterms:format, dcterms:modified, and rdfs:comment the config file records.
This is provenance: it says which exact version of each vocabulary the data next to it was validated against, and it is queryable like any other triple, through sal query --sparql, sal serve’s /sparql endpoint, or the Ontology versions sample in the SPARQL tab of sal serve --with-ui.
/sparql translates SPARQL to DuckDB SQL and only understands basic triple patterns, FILTER comparisons, DISTINCT, and LIMIT — no OPTIONAL and no ORDER BY — so the query stays inside that subset (see SPARQL Query Engine):
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>PREFIX owl: <http://www.w3.org/2002/07/owl#>PREFIX dcterms: <http://purl.org/dc/terms/>
SELECT ?ontology ?versionIRI ?format ?modifiedWHERE { ?ontology rdf:type owl:Ontology . ?ontology owl:versionIRI ?versionIRI . ?ontology dcterms:format ?format . ?ontology dcterms:modified ?modified .}ontology versionIRI format modifiedhttp://www.opengis.net/ont/geosparql# urn:sha256:6a55c78c2b38c077a31075363ef5e908fa683c61eb059a0fc9828aeb… application/rdf+xml 2026-08-11T17:21:12Zhttps://schema.org/ urn:sha256:c3a1e7c9c4b6f6b1a3f5e6d7c8b9a0e1f2d3c4b5a6978869e6fa84d… application/ld+json 2026-08-11T17:21:12ZA vocabulary the project pins but does not import, one it only checks terms against, shows up here the same as an imported one; whether an ontology’s own statements are in the table at all is a separate question from which version it validated against, and only an imported one’s are.
STAC catalog
Section titled “STAC catalog”The last thing sal build does is describe the data product as a STAC catalog under .sal/data/stac, so that a STAC client can discover it and an Iceberg reader can open it from what the catalog records.
catalog.json is a STAC Catalog named after the git project, listing triples/collection.json as a child, and
after it every STAC catalog a SAL module task handed over as a directory (see
Files a task produces): a directory under .sal/data/blobs
whose catalog.json is a STAC Catalog or Collection is linked as a child by relative path, such as
../blobs/3f9a…/catalog.json, titled by the catalog’s own title or id. The module’s files are left
where the copy put them, since their own links name their neighbours by relative path, and the link is relative
too, so the data product is one catalog tree on disk, under sal serve, and in a bucket it is uploaded to.
A sub-catalog inside a linked catalog is that catalog’s to list and is not linked again; a catalog.json that is
not STAC is warned about and skipped.
The triples/collection.json collection describes the triples table with three extensions:
- The Table extension:
table:columnsnames every column of the table with its type,table:row_countis the number of rows, the statements of the pinned vocabularies included since the catalog describes the table itself, andtable:primary_geometryisobject_geometry. - The Projection extension:
proj:codeis the CRS of the geometry column,OGC:CRS84. - The Iceberg extension, whose fields are how a reader opens the table:
{ "iceberg:catalog_type": "static", "iceberg:authorization_type": "none", "iceberg:table_id": "<git-project-name>.triples", "iceberg:metadata_location": "file:///path/to/project/.sal/data/<git-project-name>/triples/metadata/v3.metadata.json", "iceberg:format_version": 3, "iceberg:current_snapshot_id": "7847372623449923428", "iceberg:partition_spec": [ { "name": "predicate_partition", "transform": "truncate[20]", "source-id": 2, "field-id": 1000 } ]}The same metadata.json is the collection’s one asset, iceberg, with the media type application/vnd.apache.iceberg+json the extension gives Iceberg metadata.
The catalog is static: there is no catalog server, and the metadata location is a file:// URL on the machine the build ran on, which DuckDB’s iceberg_scan() or PyIceberg’s StaticTable can read directly there.
Everything about the table is read from the table itself once it is written: the columns, partition spec, format version, and metadata location from its metadata, the snapshot ID from its current snapshot, and the row count and spatial extent from a scan of its geometry column.
extent.spatial.bbox covers every geometry in the table, or the whole world when it holds none, since STAC requires a collection to state one.
extent.temporal.interval runs from the earliest snapshot still in the table to the current one, which is when the data product was built.
The collection’s title is the project node’s dcterms:title, which sal init records and sal import --title replaces, and the git project name otherwise, and its provider is the git project owner.
The catalog is regenerated whole on every build, since it names the snapshot and metadata file that build produced, and it sits under .sal/data rather than beside .sal/config.jsonld for the same reason: it is a build artifact, not a source file, and writing it into Git would leave the working tree dirty after every build.
sal clean --wipe removes it with the rest of the data product.
Published layout
Section titled “Published layout”sal push preserves .sal/data as an OCI artifact, including the pinned vocabulary documents.
The artifact metadata records the source repository and commit, which lets sal clone recreate the matching source tree and data directory.
sal upload deploys the same data to an object store.
For object stores, SAL rewrites a staged copy of the Iceberg metadata so table paths point to the remote table root.
The local .sal/data directory is not changed by upload.
An upload to a bare bucket (gs://my-bucket/) keeps the layout of .sal/data and adds the project’s config:
config.jsonld copy of .sal/config.jsonld (project node and pins)<project>/triples/ the Iceberg table (metadata/ and data/)blobs/ pinned vocabulary documents and files modules handed overstac/ the STAC catalogsal serve --data accepts a copy of that directory, a copy of .sal/data, and an upload to an explicit table root.
For examples, see Publishing Data.

