Semantic Accessibility Layer (SAL) System

The following sections outlines the SAL vision.

Problem

Open data is published in a variety of distribution formats and methods, some of which conform to established standards. These publications mostly use familiar, semi-structured formats such as CSV or JSON, which offer superficial simplicity for downstream users and applications. Nevertheless, extensible data formats, standards-based or not, lack the expressive syntax needed to relate and reuse data with contextual, interoperable semantics across diverse technology environments and domains. Data standards do contribute to interoperability and reuse, but their adoption is often limited by competing priorities, high implementation costs, legacy system constraints, and lack of incentives. The current open data ecosystem, such that it is, is stymied by its inability to foster a cycle of secondary consumption as it perpetuates low context ‘carbon copies’ of primary knowledge which undermines its use as trusted data in derivative outputs—preventing a closed loop necessary in any ecosystem.

Additionally, open data distribution methods are selected by the publisher based on intended access patterns and often shaped by resource constraints. Application Programming Interfaces (APIs) are best aligned for responsive controlled access of changing “live” datasets serving small subsets of a few key entities at a time. Conversely, static files are are typically used for infrequently updated data served from a file server or other file distribution method. Serving static files invariably offers the simplest option for the open data publisher. It requires minimal infrastructure and operational maintenance, as it avoids the need for databases, APIs, and perhaps other infrastructure. The tradeoff is that users often have to navigate fragmented access patterns just to acquire open data, creating unnecessary overhead before analysis can begin.

The compounding reality for the open data user is that questions cannot always be answered from a single dataset. Harmonizing open datasets published in various distribution formats, access, and documentation methods, places a heavy burden on the user to derive the insight needed to answer questions. Once this work is complete, there isn’t an easy way to encode and publish the essential context gathered from the process in a way that can be universally understood, related, attributed, and reused by others. As such, continuity cannot be established by open data users as a whole to answer hard questions which dictate trusted, iterative outputs.

flowchart TD 

JSON["JSON"]
CSV["CSV"]
Excel["Excel"]
ExternalDocs["External Documentation"]
ImplicitKnowledge["Implicit Knowledge"]
JSON --> WebServicesAPI
JSON --> FileServer["File Server"]
CSV --> FileServer
Excel --> FileServer
WebServicesAPI --> Harmonize1[("Harmonize")]
FileServer --> Harmonize1
ExternalDocs --context--> User1["User 1"]
ImplicitKnowledge1["Implicit Knowledge"] --context--> User1
Harmonize1 <--> User1
WebServicesAPI --> Harmonize2[("Harmonize")]
FileServer --> Harmonize2
ExternalDocs --context--> User2["User 1"]
ImplicitKnowledge --context--> User2["User 2"]
Harmonize2 <--> User2

Solution

The Semantic Accessibility Layer system (SAL) is composed of a core module specification (SAL Module) and components that build on the specification. The SAL Module specification uses an ontology (SAL Ontology, sal:) to describe a common command-line usage interface for describing data models and exposed containerized application logic that operates on them using JSON-LD (RDF). It promotes interoperability at the point of acquisition by semantically unifying data, metadata, and application logic into a portable, referenceable module—laying the groundwork for the consistent reuse of source data across diverse systems. The SAL command-line interface tool (SAL CLI) enables users to create projects using familiar software development lifecycle stages, generating knowledge products using SAL CLI through orchestration of SAL Modules. SAL enables users to co-create uniformly described portable knowledge products based on existing published sources that can serve as building blocks towards community-built knowledge networks.

flowchart TD
JSON["JSON"]
CSV["CSV"]
Excel["Excel"]
JSON --> WebServicesAPI
JSON --> FileServer["File Server"]
CSV --> FileServer
Excel --> FileServer
WebServicesAPI --> SALModule("`SAL Module(s)`")
FileServer --> SALModule
SALModule --context--> SALProject["SAL Project"]
SALProject --context--> KnowledgeProduct["Knowledge Product"]
KnowledgeProduct --context--> User1["User 1"]
KnowledgeProduct --context--> User2["User 2"]

SAL Module Command Line Interface Usage

The SAL Module specification consists of a JSON-LD based vocabulary here which describes a command line interface (CLI) usage specification for containers. When implemented by a containerized application, a SAL Module compliant container implementation provides self-describing data models and exposed application logic to produce building blocks of highly contextualized data. This data can be retrieved using the SAL Module container as a standalone command or orchestrated through SAL CLI to produce knowledge products that can be published, shared, and (re)used.

SAL Module Behaviors

Provide ontology

SAL Module expects a compliant container to provide its ontology (i.e. vocabulary) using the following command line interface usage:

docker run IMAGE sal (ontology|vocabulary|vocab)

SAL Modules MUST declare and implement at least one sal:NodeProcessor subclass in their ontology. Each sal:NodeProcessor subclass MAY add the SAL annotation properties sal:input, sal:output, sal:self to their subclass declaration. The sal:input annotation property describes the SHACL sh:NodeShape that a sal:NodeProcessor instance(s) expects to receive. Conversely, sal:output describes the SHACL sh:NodeShape that a sal:NodeProcessor instance expects to produce. Finally, sal:self describes the SHACL sh:NodeShape that an instance of the sal:NodeProcessor subclass instance should conform to.

where…

IMAGE is the name of the docker image that adheres to the SAL Module command line interface usage behavior.

Example 1 The following example displays the ontology for docker image MYIMAGE where a single sh:NodeProcessor subclass is defined ( i.e. EducationalHistoryFinder). that accepts JSON-LD node objects conforming to the SHACL shape PersonShape and produces JSON-LD node objects conforming to SHACL shape PersonAndEducationShape. Instances of EducationalHistoryFinder execute the logic to augment a schema:Person node object with their educational background when given their first name, last name, and date of birth.

$ docker run MYIMAGE sal ontology
{
    "@context": {
        "schema":"https://schema.org/",
        "sh": "http://www.w3.org/ns/shacl#",
        "xsd": "http://www.w3.org/2001/XMLSchema#",
        "sal":"https://w3id.org/datacequia/sal#",
        "owl": "http://www.w3.org/2002/07/owl#",
        "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
    },
    "@graph": [
        {
            "@id": "PersonShape",
            "@type": "sh:NodeShape",
            "sh:targetClass": {"@id":"schema:Person"},
            "sh:property": [
                {
                    "sh:path": {"@id":"schema:givenName"},
                    "sh:datatype": {"@id":"xsd:string"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1
                },
                {
                    "sh:path": {"@id":"schema:familyName"},
                    "sh:datatype": {"@id": "xsd:string"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1
                },
                {
                    "sh:path": {"@id":"schema:birthDate"},
                    "sh:datatype": {"@id":"xsd:date"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1
                }
  ]

        },
        {
            "@id": "PersonAndEducationShape",
            "@type": "sh:NodeShape",
            "sh:targetClass": {"@id":"schema:Person"},
            "sh:property" : [
                {
                    "sh:path": {"@id":"schema:givenName"},
                    "sh:datatype": {"@id":"xsd:string"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1
                },
                {
                    "sh:path": {"@id","schema:familyName"},
                    "sh:datatype": {"@id","xsd:string"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1

                },
                {
                    "sh:path": {"@id": "schema:birthDate"},
                    "sh:datatype": {"@id":"xsd:date"},
                    "sh:minCount": 1,
                    "sh:maxCount": 1

                },
                {
                    "sh:path": {"@id":"schema:alumniOf"},
                    "sh:nodeKind": "sh:IRIOrBlankNode"
                }
            ]

        },
        {
            "@id": "EducationalHistoryFinder",
            "@type": "owl:Class",
            "rdfs:subClassOf": {"@id": "sal:NodeProcessor"},
            "sal:input": {"@id":"PersonShape"},
            "sal:output": {"@id":"PersonAndEducationShape"}
        }
        ]
}

$

Run sal:NodeProcessor

A SAL Module compliant container allows a user to run a defined instance of a sal:NodeProcessor subclass (node-processing instance) declared in it’s ontology. These instances are passed to the container’s run command using the SALMODULE_TASK_INSTANCE environment variable. The sal:NodeProcessor subclass instance serves to provide context to the sal:NodeProcessor runtime logic.

Note: one simple example for adding sal:NodeProcessor properties may be to describe an external host’s connection properties that are needed by the sal:NodeProcessor run(time) logic to connect to a data source.

docker run -e SALMODULE_TASK_INSTANCE="<sal:NodeProcessor node object instance>" MYIMAGE sal run

Building on Example 1, The following example shows how an instance of sal:NodeProcessor subclass EducationalHistoryFinder is invoked from the command line.

Note: The response printed to

$ SALMODULE_TASK_INSTANCE='{"@id":"http://example.org/#pistolpete","@type":"EducationalHistoryFinder"}'
$ docker run MYIMAGE sal run -e SALMODULE_TASK_INSTANCE="${SALMODULE_TASK_INSTANCE}"
{"@id":"http://example.org/#pistolpete","@type":"schema:Person","schema:givenName":"Frank","schema:familyName":"Eaton","schema:birthDate":{"@id":"xsd:date", "@value":"1888-09-17"},"schema:alumniOf":{"@id":"http://nmsu.edu/","@type":"schema:EducationalOrganization","schema:name": "New Mexico State University"}}
$

The output is instance data belonging to the SAL Project that ran the task, so the base IRI of the JSON-LD a task emits is the project’s namespace. A relative IRI in the output resolves against the project, not against the module’s salmodule:// namespace, which names the module’s vocabulary rather than what it produces. A module that needs to name its output under a specific namespace, such as the authoritative source the data comes from, emits absolute IRIs.

A task may also produce files. An IRI written with the file:/// scheme, such as {"@id": "file:///tmp/test.txt"}, names a file inside the container that the SAL Project copies out while the task runs, stores by the SHA-256 digest of its contents, and refers to as urn:sha256:<digest> in place of the path, with the file’s name recorded as rdfs:label, its modification time as dcterms:modified, its digest as owl:versionIRI, its path under the project’s blob store as dcterms:identifier, and the module as dcterms:source. Any other statement the task makes about the file:/// IRI is kept under the new name. An IRI ending in a slash, such as {"@id": "file:///out/catalog/"}, names a directory that is copied whole and stored under <digest>/ with its files verbatim, so that a Zarr store or STAC catalog is readable in place under a name no other module’s output can collide with. Every copy is recorded in the blob store’s prov.jsonld. Since the copy happens as soon as the line naming the path is read from stdout, a task must write a file, or every file in a directory, completely before it prints the node that refers to it. Each path is copied once; further references to it are skipped.

When the task’s class declares a salmodule:stdoutShape, the SAL Project validates each line of stdout against that sh:NodeShape as it is read, with the module ontology’s @context injected first. The first line that does not conform stops the container and fails the run with the shape’s sh:message; nothing the task wrote is kept. A salmodule:stdoutShape may use any SHACL Core constraint except sh:languageIn, sh:uniqueLang, sh:disjoint, sh:qualifiedValueShapesDisjoint, and sh:sparql.

SAL Module Protocol Scheme

SAL Modules have a protocol scheme (sal:moduleProtocolScheme) to define how SAL Projects (de)reference SAL Module ontologies in their projects. The following diagram illustrates the process of dereferencing a SAL Module protocol scheme IRI referenced in a JSON-LD document to obtain it’s ontology.

sequenceDiagram

actor User
participant JSONLDProcessor
participant Docker
participant Git
participant RemoteGitRepository 
participant LocalGitRepository

User ->> JSONLDProcessor: Process JSON-LD Document
activate JSONLDProcessor
JSONLDProcessor ->> JSONLDProcessor: parse json-ld
activate JSONLDProcessor
JSONLDProcessor ->> JSONLDProcessor: dereference salmodule://[HOST/]USER/REPO#35;

JSONLDProcessor ->> Git: git clone [HOST/]USER/repo
activate Git
Git ->> RemoteGitRepository: 
RemoteGitRepository -->> Git: 
Git ->> LocalGitRepository: 
LocalGitRepository -->> Git: 
Git -->> JSONLDProcessor: 
deactivate Git 

JSONLDProcessor ->> Docker: docker build 
activate Docker
Docker ->> LocalGitRepository: process Dockerfile in repo base directory
LocalGitRepository -->> Docker: 
Docker -->> JSONLDProcessor: <SAL Module Image>
deactivate Docker

JSONLDProcessor ->> Docker: run <SAL Module Image> sal ontology
activate Docker
Docker -->> JSONLDProcessor: ontology from stdout
deactivate Docker
JSONLDProcessor -->> User: 


deactivate JSONLDProcessor