Using SAL Modules

A SAL Module is a git repository with a Dockerfile in its root whose container implements the SAL Module command line interface. Referencing one from your project’s RDF lets sal run run it and fold the RDF it produces into your data product.

Reference a module

A module is referenced through the salmodule:// protocol scheme. Declare it as a prefix and type an instance with one of the classes the module’s ontology defines:

@prefix salmodule: <https://w3id.org/sal/cgs-earth/sal-module-spec/salmodule#> .
@prefix states: <salmodule://github.com/cgs-earth/python-geoconnex/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<GeoconnexStates> a states:GeoconnexReferenceFeatureStates ;
    a salmodule:NodeProcessor ;
    states:maxRetries "5"^^xsd:integer .

The prefix takes the form salmodule://[HOST/]OWNER/REPO/. HOST is optional and defaults to github.com.

The prefix must end in a slash

A module’s ontology declares its classes as relative IRIs, such as "@id": "GeoconnexReferenceFeatureStates". SAL resolves them against the prefix you declared, and only a slash-terminated prefix places them underneath the module:

# correct
@prefix states: <salmodule://github.com/cgs-earth/python-geoconnex/> .
# states:GeoconnexReferenceFeatureStates and the ontology's own term both become
#   salmodule://github.com/cgs-earth/python-geoconnex/GeoconnexReferenceFeatureStates

# wrong
@prefix states: <salmodule://github.com/cgs-earth/python-geoconnex#> .
# relative IRI resolution drops the last path segment, so the ontology's term becomes
#   salmodule://github.com/cgs-earth/GeoconnexReferenceFeatureStates
# which no longer matches the term the prefix expands to

A prefix that does not end in a slash reports every module term as undefined even though the module and its ontology are fine, so reach for the slash-terminated form whenever a module’s terms unexpectedly fail to validate.

Configure the task

A task is configured in RDF, with the properties the module’s own vocabulary defines. states:maxRetries above is one of them, so the module decides what a valid task instance looks like and sal validate reports a typo in a configuration property the same way it reports a typo in a class.

SAL serializes those properties into the JSON-LD node object that salmodule:runCommand is invoked with. The instance above is passed as:

{"@id":"https://example.org/GeoconnexStates","@type":"GeoconnexReferenceFeatureStates","maxRetries":{"@value":"5","@type":"xsd:integer"}}

The rules SAL follows when writing it:

Only properties declared as vocabulary in the module’s own ontology become keys. Everything else the project says about the instance, such as an rdfs:label or a schema:name, is left out, since only the module knows what configures it.
Keys and IRIs are written the way the module’s ontology writes them: a term of the module’s vocabulary becomes its relative name, and any other IRI is shortened with whichever prefix the ontology’s @context binds it to.
A typed literal becomes a {"@value": ..., "@type": ...} object and a language-tagged literal becomes a {"@value": ..., "@language": ...} object. A plain string is written as a string.
A property used more than once becomes an array.
A blank node is inlined as a nested object, so structured configuration can be expressed with the usual turtle [ ... ] syntax. A named node stays a {"@id": ...} reference.

Which ontology property describes what

Two properties of the SAL Module ontology are involved, and it is worth keeping them apart:

salmodule:runCommand describes the invocation the task instance is an input to, and so is the property that documents the JSON-LD node object above. Its skos:note is what says the value must be a JSON-LD representation of the salmodule:Task subclass instance being invoked.
salmodule:taskInstanceEnvVar only defines a constant: the name of the environment variable that value is carried in. A module declares it on its own ontology, and it defaults to SALMODULE_TASK_INSTANCE.

What SAL does with it

Validation dereferences the salmodule:// prefix so that the module’s terms are checked like any other vocabulary. SAL clones the repository, builds the Dockerfile in its root, and runs docker run IMAGE salmodule ontology to read the module’s vocabulary. A typo in a module term is reported the same way a typo in a schema: term is:

data/module.ttl:5: undefined term states:GeoconnexReferenceFeatureStatess

sal build commits the task’s configuration into the data product without running anything. sal run then runs each task instance with docker run -e SALMODULE_TASK_INSTANCE=... IMAGE salmodule run, once the worktree is fully committed and the table’s latest snapshot was built from the commit HEAD points at. An instance is only run when it is typed as a SAL Module task, either because the project’s RDF types it as a salmodule:NodeProcessor, salmodule:NodeProducer, salmodule:NodeConsumer, or salmodule:Task, or because the module’s own ontology declares its class as a subclass of one of those.

A task writes newline delimited JSON to stdout. SAL injects the module ontology’s @context into each node so that its keys resolve to the module’s vocabulary, converts the result to RDF, and commits it on top of the last build as a new snapshot. Those triples land in the Iceberg table alongside the triples from your source files.

If a task emits salmodule:Error nodes instead of data, the run fails with the messages the module reported.

Importing a module’s ontology

Referencing a module checks its terms and lets sal run run its tasks, but the vocabulary itself stays outside the data product. Import the module to carry it too:

sal import salmodule://github.com/cgs-earth/python-geoconnex

sal build dereferences the import the same way it dereferences a salmodule:// prefix, by cloning the repository, building the image, and running salmodule ontology, and merges the resulting triples into the table. The module is written without a trailing slash here, since the import names the module rather than the vocabulary base its terms resolve against.

See sal import.

Modules recorded in the table

Every module a build downloaded, whether it was dereferenced for its vocabulary or run as a task, is written to the Iceberg table metadata as a JSON list under the sal.salmodules property:

["salmodule://github.com/cgs-earth/python-geoconnex"]

The list is rewritten on every build and run, so dropping a module from your RDF drops it from the table too. sal query --info properties prints it, and the Modules tab of sal serve --with-ui offers each entry as a chip and as an autocompletion so a module the table was built from can be inspected without typing its URI.

Requirements

A reachable docker daemon. SAL talks to whatever DOCKER_HOST points at, defaulting to the local socket.
git on PATH, used to clone the module repository.

Each module is cloned and built at most once per sal invocation, and its image is tagged with the git commit hash of the repository it was built from. A later invocation looks that tag up before doing anything: when the commit the project pins in .sal/config.jsonld still has its image on the docker daemon, the module is neither cloned nor built again, and its container runs from the existing image. Removing the image with docker rmi brings back the clone and build.

A dereferenced module ontology is pinned like any other vocabulary the project resolves against: its JSON-LD is stored under .sal/data and recorded in .sal/config.jsonld, so a later build validates against the same version of the module’s vocabulary without rebuilding the module to ask for it again. --no-cache clones every referenced module again, ignoring the pinned image, and re-pins the ontology each one reports. See Pinned vocabularies.

Materialization is not pinned: sal run runs each task every time it is invoked, since a task’s output is data rather than vocabulary.

Developed by