Skip to content

Commit

Permalink
Add pipeline documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Ethan Grahn committed Nov 17, 2022
1 parent 22c5e13 commit c9b38a0
Showing 1 changed file with 80 additions and 46 deletions.
126 changes: 80 additions & 46 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,95 +6,112 @@
This repository houses code behind the Network Linked Data Index (NLDI) API [(Swagger Docs)](https://labs.waterdata.usgs.gov/api/nldi/swagger-ui/index.html). The NLDI is hosted as part of the [USGS Waterdata Labs](https://labs.waterdata.usgs.gov/index.html), a set of new capabilities being developed by the USGS Water Mission Area.

## Table of Contents
* [Public API](#public-api)
* [The root is {host}/api/nldi](#the-root-is-hostapinldi)
* [Up/Down Stream navigation](#updown-stream-navigation)
* [Up/Down Stream data](#updown-stream-data)
* [Query Parameters](#query-parameters)
* [Other Endpoints](#other-endpoints)
* [Development](#development)
* [Dependencies](#dependencies)
* [Docker Compose](#docker-compose)
* [Environment variables](#environment-variables)
* [Testing](#testing)

- [Public API](#public-api)
- [Top Level Path](#top-level-path)
- [Up/Down Stream Navigation](#updown-stream-navigation)
- [Up/Down Stream Data](#updown-stream-data)
- [Query Parameters](#query-parameters)
- [Other Endpoints](#other-endpoints)
- [Development](#development)
- [Dependencies](#dependencies)
- [Docker Compose](#docker-compose)
- [Environment Variables](#environment-variables)
- [Testing](#testing)
- [Pipelines](#pipelines)
- [Test](#test)
- [Build](#build)
- [Deploy](#deploy)

## Public API
The services are accessed via an http GET request. All output is generated as JSON and GeoJSON.

### The root is {host}/api/nldi
Both a test and a production are exposed to the public:
<https://labs-beta.waterdata.usgs.gov/api/nldi/linked-data> is the test root.
<https://labs.waterdata.usgs.gov/api/nldi/linked-data> is the production root.
The services are accessed via an HTTP GET request. All output is generated as JSON and GeoJSON.

### Top Level Path

This endpoint will give you the valid dataSource names for the other endpoints. There is also a health check at /about/health and version information at /about/info.
The root path for the NLDI services is `/api/nldi` and follows the hostname under which it is hosted. For example, the QA public endpoint is <https://labs-beta.waterdata.usgs.gov/api/nldi/> and the production endpoint is <https://labs.waterdata.usgs.gov/api/nldi/>. The root path will not return any information. Instead, utilize the sub-paths mentioned in the [Swagger documention](https://labs.waterdata.usgs.gov/api/nldi/swagger-ui/index.html) or the examples below.

The [/api/nldi/linked-data](https://labs.waterdata.usgs.gov/api/nldi/linked-data/) endpoint will give you the valid data source names for the other endpoints. There is also a health check at `/about/health` and version information at `/about/info`.

In general, the API uses hypermedia to help discover options from a given endpoint. A summary of these options follows.

### Up/Down Stream navigation
### Up/Down Stream Navigation

/{featureSource}/{featureSourceId}/navigation/{navigationMode} where:
* `{featureSource}` identifies the source used to start navigation:
* The `comid` `{featureSource}` starts the navigation from an NHDPlus comid
* any of the network linked feature sources (listed at /)
* `{featureSourceId}` the NHDPlus comid or feature from which to start the navigation
* `{navigationMode}` is the direction and type of navigation:
* `DD` is `D`ownstream navigation with `D`iversions
* `DM` is `D`ownstream navigation on the `M`ain channel
* `PP` is `P`oint to `P`oint navigation (the `stopComid` query parameter is required and must be downstream of the `{comid}`)
* `UM` is `U`pstream navigation on the `M`ain channel
* `UT` is `U`pstream navigation including all `T`ributaries

### Up/Down Stream data

- `{featureSource}` identifies the source used to start navigation:
- The `comid` `{featureSource}` starts the navigation from an NHDPlus comid
- any of the network linked feature sources (listed at /)
- `{featureSourceId}` the NHDPlus comid or feature from which to start the navigation
- `{navigationMode}` is the direction and type of navigation:
- `DD` is `D`ownstream navigation with `D`iversions
- `DM` is `D`ownstream navigation on the `M`ain channel
- `PP` is `P`oint to `P`oint navigation (the `stopComid` query parameter is required and must be downstream of the `{comid}`)
- `UM` is `U`pstream navigation on the `M`ain channel
- `UT` is `U`pstream navigation including all `T`ributaries

### Up/Down Stream Data

/{featureSource}/{featureSourceId}/navigation/{navigationMode}/{dataSource} where:
* `{featureSource}` identifies the source used to start navigation
* `{featureSourceId}` the NHDPlus comid or other feature from which to start the navigation
* `{navigationMode}` is the direction and type of navigation
* `{dataSource}` is the abbreviation of the data source to return

- `{featureSource}` identifies the source used to start navigation
- `{featureSourceId}` the NHDPlus comid or other feature from which to start the navigation
- `{navigationMode}` is the direction and type of navigation
- `{dataSource}` is the abbreviation of the data source to return

### Query Parameters
Navigations accept query parameters to further refine/restrict the navigation being requested
* `f=json` if an html media type is requested explicitely, a blank html page is returned. `f=json` will override this html accept header.
* `distance={dist}` **REQUIRED** limits the navigation to `{dist}` kilometers from the starting point
* `stopComid={stopid}` for use with `PP` navigation between the `{featureSourceId}` and `{stopid}`
* (only applicable to NHDPlus comid navigation and the `{stopid}` must be downstream of the `{featureSourceId}`)

Navigations accept query parameters to further refine/restrict the navigation being requested.

- `f=json` if an html media type is requested explicitely, a blank html page is returned. `f=json` will override this html accept header.
- `distance={dist}` **REQUIRED** limits the navigation to `{dist}` kilometers from the starting point
- `stopComid={stopid}` for use with `PP` navigation between the `{featureSourceId}` and `{stopid}`
- (only applicable to NHDPlus comid navigation and the `{stopid}` must be downstream of the `{featureSourceId}`)

### Other Endpoints
The NLDI includes additional helper endpoints that will be documented here at a later date. See the [swagger documentation](https://labs.waterdata.usgs.gov/api/nldi/swagger-ui/index.html) and [NLDI landing page](https://labs.waterdata.usgs.gov/about-nldi/index.html) for more.

The NLDI includes additional helper endpoints that will be documented here at a later date. See the [Swagger documentation](https://labs.waterdata.usgs.gov/api/nldi/swagger-ui/index.html) and [NLDI landing page](https://labs.waterdata.usgs.gov/about-nldi/index.html) for more.

## Development

This is a Spring Batch/Boot project. All of the normal caveats relating to a Spring Batch/Boot application apply.
In general, do not run this project via Docker locally, since it places everything under root ownership.
Rather, start up the demo db and create an application.yml file as described below, then run the project from your IDE.

### Dependencies

This application utilizes a PostgreSQL database.
[nldi-db](https://github.com/internetofwater/nldi-db) contains everything you need to set up a development database environment. It includes data for the Yahara River in Wisconsin.

### Docker Compose

This project includes a Docker-Compose file with all necessary variables predefined.
First, start the demo database by running:

```sh
```shell
docker-compose up -d nldi-db
```

Then, build and start the NLDI services by running:

```sh
```shell
docker-compose up nldi-services
```

These test services will be accessible at <localhost:8080/nldi>. \
If you would like to build these images using a mirror url simply set a `DOCKER_MIRROR` environment variable or include
it as a build argument.

### Environment variables
### Environment Variables

To run the project (connecting to a separately running db instance) you will need to create the file application.yml in the project's root directory and add the following (normal defaults are filled in):

```yaml
nldiDbHost: localhost
nldiDbPort: 5437 #Or whatever port you map it to
nldiDbUsername: [dbUserName] #See nldi-db project .env file 'NLDI_READ_ONLY_USERNAME'
nldiDbPassword: [dbPassword] #See nldi-db project .env file 'NLDI_READ_ONLY_PASSWORD'
nldiDbName: [dbName] #See nldi-db project .env file 'NLDI_DATABASE_NAME'
nldiDbUsername: dbUserName #See nldi-db project .env file 'NLDI_READ_ONLY_USERNAME'
nldiDbPassword: dbPassword #See nldi-db project .env file 'NLDI_READ_ONLY_PASSWORD'
nldiDbName: dbName #See nldi-db project .env file 'NLDI_DATABASE_NAME'
nldiProtocol: http
nldiHost: owi-test.usgs.gov:8080
nldiPath: /test-url
Expand All @@ -108,6 +125,7 @@ spring.security.user.password: changeMe
```
### Testing
This project contains JUnit tests. Maven can be used to run them (in addition to the capabilities of your IDE).
To run the unit tests of the application use:
Expand All @@ -121,3 +139,19 @@ To additionally start up a Docker database and run the integration tests of the
```shell
mvn verify
```

### Pipelines

This project has automated pipelines for test, build, and deploy.

#### Test

The test pipeline is run when a pull request is created and consists of two parts. The first step is linting using the Spotless Maven plugin. This validates that any code changes fit within certain style guidelines to maintain readability and consistency. The second stage runs all unit and integration tests. If both of these stages pass, a pull request is eligible to be merged.

#### Build

The build pipeline happens internally to the USGS and are triggered manually. The `docker build` command is run to construct a Docker container after which it is pushed to an internal container registry.

#### Deploy

Similar to the build pipeline, deploys are internal and triggered manually. The infrastructure components are managed with this pipeline and retrieve the Docker container from the previously mentioned container registry.

0 comments on commit c9b38a0

Please sign in to comment.