Geoconnex API

Feedback on requests and responses

Introduction

Geoconnex is a decentralized metadata catalog that incorporates information about data collected by any organization publishing water data, including the spatial, temporal, and topical subjects of that data.

We are developing an API to allow programmatic access to this catalog

• for potential new data publishing organizations to enrich their own metadata

• for data analysts and tool developers to discover water metadata, like a specialized Google search for water data.

Thank you for participating etc. We are asking for feedback on the API for geoconnex. Geoconnex is a knowldege graph that aims to allow data users to discover relationships between real-world hydrologic features, cataloging features, organizational monitoring locations with data about them, and reference locations that serve as organizational monitoring locations common across multiple organizations.

What I’ll do for you today is introduce the main use cases we imagine for the geoconnex API, and then go through a number of endpoints we are in the early design phases of. This is truly the first draft so your input could have a dramatic effect on the production API. Thus, rather than presenting fully specified API HTTP requests, and json payloads for requests and responses, I will describe the basic structure of the queries we aim to make possible, and the inputs and outputs that we are thinking about implementing.

Introduction

What kind of water data do you work with?

How do you normally find water data?

What, if any, challenges do you have finding water data from particular or unfamiliar sources?

Introduction

Geoconnex itself will be a metadata catalog that includes minimally standardized metadata about all sites (eg gages, wells, dams, public water systems, water quality sample stations, etc.) that all participating organizations (eg federal, state, local, Tribal, NGO, academic) publish data about. In this diagram, we are focusing on the data publisher, reference.geoconnex.us, the data user, and api.geoconnex.us. In the geoconenx system, data publishers publish metadata about datasets that they collect that are about certian common reference features across many data publishers.These common features we call “reference features”. The geoconnex system aggregates the metadata that data publishers provide, and provides centralized access to the resulting metadata catalog. The geoconnex API will allow programmatic access to this metadata catalog. The Geoconnex API will thus provide data discovery services, but not necessarily observation/model data access directly. It will tell users that data on certain subjects exist, and where to find that data, like a library catalog. It will not directly provide the observed values. However, It will point users to relevant datasets with as much useful metadata as possible. First I’ll introduce you to the two high level use cases we imagine this API meeting, and then show you mockups of the API endpoints, their inputs, and responses.

Outline

• User Story - Data Publisher
• User Story - Data Analyst/Tool Developer
• Endpoints, requests, and responses

The Geoconnex API we have designed around two main user groups, data publishers looking to get their datasets represented in the system, and data analysts or tool developers who need to discover water datasets across all organizations that might be relevant to their questions. I’ll then give a deeper dive to ask for your feedback about our designes for API endpoitns, and their requests (or inputs) and their responses (or outputs)

User Story - Water Data Publisher

• “As a water data publisher…

• I want to know all locations from all organizations where data is currently being collected that are near my own monitoring locations

• So that I know whether to submit reference features to geoconnex.us, and so I know which reference features to tag my own data with when I publish it.

User Story - Water Data Analyst/Tool Developer

• “As a water information tool/product developer…

• I want to know all locations from all organizations where data types relevant to my questions are collected, and where that data can be accessed

• So that I can build my tool or conduct my analysis using as much relevant data as possible

Top-level Endpoints

https://api.geoconnex.us

• Services to discover and filter water datasets by:

  Space/Geometry	Measurement/Modeling Method
  Organization/ data provider	Temporal resolution
  Site Type (eg stream, well, dam)	Temporal coverage
  Parameter/Variable/Observed Property	Feature of Interest

THe top level endpoint for the geoconnex APIs will be api.geoconnex.us. There will be several endpoints under this one. Most importantly, there will be a suite of services that allow users to discover and filter water datasets by these kinds of attributes. Are there other kinds of filters that might be desireable?

Top-level Endpoints

https://api.geoconnex.us…

• /processes ¹
• /locationTypes ²
• /FeatureCollections ³
• /Features/{catalogingFeatureType} ⁴
• /providers ⁵
• /observedProperties ⁶
• /methodTypes ⁷
• /methods ⁸
• /timeSpacing ⁹

example codelist: methods

id	provider	name	description	methodType	url
noaa-ahps	ahps	ahps	NOAA Advanced Hydrological Prediction Service	forecast	https://water.weather.gov/ahps/about/about.php
usgs-streamflow	nwis	WSP-2175	Streamflow measurement and computation	in-situ observation	https://pubs.usgs.gov/wsp/wsp2175/wsp2175.pdf
usgs-groundwater_level-tape	nwis	GWPD1	Measuring water levels by use of a graduated steel tape	in-situ observation	https://pubs.usgs.gov/tm/1a1/pdf/GWPD1.pdf

THe first endpoint, processes, will describe the actual metadata search and retrieval services. The other endpoints are all annotated codelists for filters that can be used to subset all of the sites. After reviewing each one, do you have any opinions on if any of these are not useful? If there are other such codelists that might be useful. Note how most codelists include an id, a name, a description, and in some cases a URL. Is this enough information? Should more information be provided about each code

Processes

Data discovery tools will follow the OGC-API Processes standard, and thus we refer to them as processes. The following tools are proposed:

process name	description
referenceMyLocations	POST a geospatial dataset of your own locations, find candidate geoconnex reference locations that may correspond to them already
findFeatures 10	GET or POST query parameters to find features of relevance
navigateFeatures 11	GET all hydrologically relevant monitoring features for a given longitude and latitude or feature identifier
findDatasets 12	POST an array of feature identifiers and an array of query parameters to find relevant dataset metadata]

/processes Data publisher use case

• As a data publisher, would use referenceMyLocations to see if any existing locations in the database exist that are likely the same site/ real-world object as sites I want to publish data about. I could then add links in my metadata that I publish to ensure others know I have data about the same site.

referenceMyLocation Inputs:

• input data: a geospatial dataset with at least an ID field

• locationType: from /locationType codelist (eg “gage”, “well”, “dam”)

Note that throughout we are showing the conceptual input and outputs rather than fully specifying for example, JSON structures, given where we are in the design process. Just explain that the endpoint as currently designed takes 2 inputs: a geospatial dataset and text locationType corresponding to a member of the locationType codelist from the top level endpoints slide.

/processes Data publisher use case example

POST to /processes/referenceMyLocations

Input:

• locationType: “gage”

• input data:

This is meant to show an example input, and the next slide will visualize the example output. Note that map is zoomable and clickable. Let them know this is just for visualization pruposes for the type of input that would be provided. Just explain, and move on to next

/processes Data publisher use case example

POST to /processes/referenceMyLocations

Output:

• spatial dataset with fields: uri, name, input_id, match_distance_m

Note that the map is interactive as before. At this point, ask if they have, concerns, suggestions at a high level for:

1. other inputs they think should be there
2. other fields that the output should have
3. formats we should think about supporting for the inputs and outputs
4. other services that could be useful to a data publisher

/processes Data user imagined workflow

1. use findFeatures discover locations within a bounding box, radius around a point, or arbitrary spatial polygon, or relevant to reference feature (eg a river or aquifer) or cataloging feature (eg a an administrative boundary or HUC)

   • receive a geospatial dataset, including attribute id populated with geoconnex http URIs (e.g. https://geoconnex.us/usgs/monitoring-location/1000001)

2. If desired, use navigateFeatures to find all sites downstream and/or upstream of a site identified by a geoconnex http URI or a latitude/longitude

3. use findDatasets, given a list of URIs and query parameters for provider, observedProperty, period of record, etc., be given links to datasets relevant to your question that are about the locations found by findFeatures and/or navigateFeatures calls

go through imagined workflow, ask if they have questions, concerns, suggestions at a high level for other services, or a different configuration of services, that could be useful to a data user. Then explain that we’ll go through what we have designed in more detail to get some finer grained feedback.

/processes/findFeatures use case

• As a data user, I would use findFeatures to discover all real-world features that organizations of interest have published metadata about, fitltered to only my area of interest

/processes/findFeatures inputs

input	number of elements	example or description
locationTypes	min:0, max: inf	["gages", "wells"]
providers	min:0, max: inf	NULL 13, or ["nwis","rise"]
catalogingFeatures	min:1, max: inf	{"hu02:"14", "state":["CA","AZ"]}
hydrologicFeature	min:0, max: inf	["https://geoconnex.us/ref/mainstems/1"]
polygon	min:0, max:1	any multipolygon or polygon geometry
bbox	min:0, max:1	xmin, xmax, ymin, ymax
radius	min: 0, max: 1	lat, lon, distance in km

Explain that this is a table describing all imagined possible inputs. THe only required input is that at least 1 cataloging feature should be specified, although moultiple can be. Explain that across inputs, the implied boolean operator is “and”, but within inputs, if you put multiple, the inplied operator is “or”. Ask If there are questions or concerns about any of these inputs. Ask if any other inputs would be desirable.

/processes/findFeatures output

geospatial dataset of all locations meeting input field criteria, with fields:

• uri eg https://geoconnex.us/foo

• name (by provider) eg

  • name: "colorado river at bridge x", provider: "nwis"

  • name: "station WQX1234", provider: "storet"

• locationType (eg "stream")

• Cataloging Features uri for every catalogingFeatureCollection eg:

  • hu02: https://geoconnex.us/ref/hu02/14

  • county: https://geoconnex.us/ref/counties/06025

• all relevant Hydrologic Features eg: mainstem: https://geoconnex.us/ref/mainstems/29559

Say this describes the attributes available in an output dataset. If they ask if there are coordinates, say it would be some kind of geospatial output, whether the features end up being points or lines or polygons. Once done, ask if they have any concerns about these outputs. If they wish anything were different, or if they think more or less attributes should be there, or if they think any important attributes are missing.

/processes/navigateFeatures use case

• As a data user, I would use navigateFeatures to discover all real-world features that organizations of interest have published metadata about, that are hydrologically relevant (eg upstream or downstream of) to a point that I am interested in.

/processes/navigateFeatures inputs and outputs

inputs

• uri or pointLocation (lat/lon)
• upstream: null, tributaries, or mainstem
• downstream: null, diversions, or mainstem
• distance: integer (km)
• locationType: text¹⁴
• provider: text¹⁵
• catalogingFeatures: array of uri¹⁶

outputs

• mainstem geospatial feature
• tributaries geospatial feature collection
• diversions geospatial feature collection
• relevant geospatial dataset, with attributes:

  • uri: https://geoconnex.us/foo

  • hydro_relation:

    • “upstream mainstem/tributary” or “downstream mainstem/diversion”

    • distance_km: integer

  • all attributes as from findFeatures

The main use case of this process is to take a single uri (which is an already registered location in geoconnex) or point, and delineate the relevant hydrology upstream and downstream, and then to retrieve all other sites in geoconnex that are hydrologically relevant. The output will include 4 objects: 3 geospatial datasets indicating the mainstem, tributary, and diversion streams relevant to the query point, and then a dataset of all indexed geoconnex locations, with all available attributes that would be found in the preivously introduced findLocations Process. Do you have any questions or concerns about the inputs or outputs? If things jsut aren’t clear, let me go through the example ont he next two slides

/processes/navigateFeatures example inputs

inputs:

lat: 35.45 lon: -105.14 upstream: tributaries downstream: mainstem distance: 3000 catalogingFeatures: https://geoconnex.us/ref/states/35

Here we show an input at a point, going upstreamm including tributaries and downstream only the mainstem, a maximum distance of 3000km, and we want to filter the results to only the state of New Mexico. THe map visualizes the inputs.

/processes/navigateFeatures example outputs

outputs:

Here we show a map which visualizes the output. Do note the actual output would likely be geojson, with a map like this as a possible html output. Feel free to zoom in. Note that we dont include the distance output attribute in this demonstration. Do you have any questions or concerns about the inputs or outputs? Do you have any suggestions for ways they could be different?

/processes/findDatasets use case

• As a data user, I would use findDatasets to discover all datasets about topics I am interested about, that are relevant to sites a set of sites in the geoconnex system that I am already interested in.

• Inputs: a list of geoconnex URIs from findFeatures and/or navigateFeatures, and query parameters to filter datasets relevant to my topic of interest

• Outputs: a table or array of dataset metadata, including which geoconnex URI datasets are relevant to.

read through use case. Explain that the next slide will go into more detail about the inputs and outputs, and that we’d lvoe their feedback on specifically the inputs and outputs that woould be most useful to add, or that are not useful and could be reomved or changed.

/processes/findDatasets inputs and outputs

inputs

• uri: array (many URIs)
• providers: null(all) or array of 1 or more providers
• observedProperties: null(all) or array of 1 or more observed Property codes
• methodTypes or methods¹⁷
• timeSpacing: the maximum (lowest resolution) timespacing desired
• minYear: the earliest year you want returned datasets to have coverage over
• maxYear: the latest year you want returned datasets to have coverage over

outputs

An table of dataset metadata with the following fields

• url: url for where each dataset lives
• about_uri: the URI for the geoconnex location each dataset is about
• provider: code from /providers
• observedProperty: code from /observedProperties
• methodType: code from /methodType
• method: code from /methods
• timeSpacing: code from /timeSpacing
• minYear: integer
• maxYear: integer
• conformsTo: a url to a website describing the API, data model, or data dictionary relevant to the specific dataset

To reiterate, our idea was that you would use one or both of findFeatures and navigateFeatures to get a list of URIs. You could then pass those URIs and the dataset query parameters to retrieve a list of datasets. Having already procured information about the locations, you would input desired dataset attributes including observed property, methods, and temporal coverage and resolution. The set of metadata you would retrieve would include that information, including which URI the dataset is about, as well as a special property, conformsTo, which would link out to a web resource that descirbes how to use that dataset. Since geoconnex is an index, it is open to all source formats. Depending on the dataset, conformsTo might point to an API or data model documentation page, a data dictionary file, or narrative documentation of the dataset.

Do you ahve any questions or concerns about the inputs and outputs of the FindDatasets Process?

Thinking back on the whole workflow, now that you’ve seen the design from end to end, any additional questions, concerns or suggestions?

THANK YOU

Feel free to continue to peruse this presentation

Send any additional feedback to konda@lincolninst.edu

Footnotes

1. an array of API endpoints in the OGC-API Processes standard for more specific queries that may use the below as query parameters

2. a nested json array of types of locations available and their definition for each (e.g. [{“id”:“dam”, “name”:“dam”, “description”: “a structure creating an impounded body of water on a stream”)},{“name”:“stream”, “description”: “a flowing body of water on the surface”}])

3. types of hydrologic (eg river, aquifer) and cataloging (e.g. HUC, catchment, county, state, municipality) features that data may be about or relevant to

4. an array of the names and identifiers the reference features of a given type

5. an array of identifiers, names, and URLs for data publication systems and their parent organizations (eg, nwis, National Water Information System, waterdata.usgs.gov, usgs.gov)

6. an array of identfiers, names, and provider identifiers for observed properties (also often known as parameters, variables, data types) eg {“id”:“inflow_lake”, “name”:“Lake/Reservoir Inflow”,“provider”:“nwis”}.

7. an array of identifiers, names, and definitions for broad categories of methods and their definitions (eg “id”:“obs”,“name”:“in-situ observation”,“definition”:“observation from an in-situ sensor or sample from a site visit”). Also include “remote sensing”, “estimation”, “simulation model”, “forecast”, “statistical summary”.

8. an array id ids, names, descriptions, and links to documentation for specific data production methods/sensor eg {“id”:“noaa-ahps” ,“provider”:“noaa”, “name”:“ahps”, “description”:“NOAA Advanced Hydrologic Prediction Service River Forecast Model”,“methodType”:“forecast”, “url”: “https://water.weather.gov/ahps/about/about.php”}

9. an array of time spacings of datasets available from sites eg [“unknown”,“intermittent”,“discrete”,“event”,“1 second”, “15 minute”, “1 day”, “1 year”]

10. similar to NWIS Site Service

11. similar to NLDI or upstream/downstream EPA RAD/WATERS

12. similar to whatNWISdata function from dataRetrieval

13. retrieves from all providers

14. to pre-filter results by location type

15. to pre-filter results by provider

16. useful if you want to restrict hydrologic navigation to certain non-hydrologic boundaries

17. to avoid conflicts between specifying a certain in-situ method and the methodType ’forecast”, for example