Lateral

.github/workflows/build-docker-image.yaml

@@ -6,14 +6,14 @@ on:
     branches:
       - 'main'
     paths:
-      - 'Dockerfile'
+      - 'docker/runner/Dockerfile'
     tags:
       - 'v*'
   pull_request:
     branches:
       - 'main'
     paths:
-      - 'Dockerfile'
+      - 'docker/runner/Dockerfile'
 
 env:
   REGISTRY: ghcr.io
@@ -46,7 +46,7 @@ jobs:
       - name: Build and push Docker image
         uses: docker/build-push-action@f2a1d5e99d037542a71f64918e516c093c6f3fc4
         with:
-          context: .
+          context: docker/runner
           push: ${{ github.event_name != 'pull_request' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}

README.md

@@ -16,69 +16,71 @@ See the [Documentation](https://smnorris.github.io/bcfishpass/) for details.
 ## General requirements
 
 - bash
-- GDAL (tested with v3.6)
-- a PostgreSQL / PostGIS database (tested with v14/v3.3)
-- Python (tested with v3.11.0)
+- GDAL
+- PostgreSQL / PostGIS
+- Python
 - [bcdata](https://github.com/smnorris/bcdata)
 - [fwapg](https://github.com/smnorris/fwapg)
 - [bcfishobs](https://github.com/smnorris/bcfishobs)
 
-## Setup / Usage
+## Development setup
 
 `bcfishpass` is a collection of shell/sql/Python scripts. To download and use the latest:
 
     git clone https://github.com/smnorris/bcfishpass.git
     cd bcfishpass
 
-Install required tools using your preferred method. For local development, `conda` can be simplest:
+All scripts presume the path to an existing PostGIS enabled database is defined by the environment variable `$DATABASE_URL`:
 
-    conda env create -f environment.yml
-    conda activate bcfishpass
+    export DATABASE_URL=postgresql://postgres@localhost:5432/bcfishpass
 
-A Docker image is also provided:
+Install all other required tools/dependencies using your preferred package manager or via Docker.
+A `conda` environment and a `Dockerfile` are provided:
 
-    docker pull ghcr.io/smnorris/bcfishpass:main
+#### Conda
 
-If the database you are working with does not already exist, create it:
+    conda env create -f environment.yml
+    conda activate bcfishpass
+    jobs/<script>
 
-    createdb bcfishpass
+#### Docker
 
-All scripts presume that the `DATABASE_URL` environment variable points to your database. For example:
+    docker compose build
+    docker compose up -d
+    docker compose run --rm runner jobs/<script>
 
-    export DATABASE_URL=postgresql://postgres@localhost:5432/bcfishpass
+Docker is configured to write the database to `postgres-data` - even if containers are deleted, the database will be retained here.
+If you have shut down Docker or the container, start it up again with this command:
 
-Load FWA:
+    docker-compose up -d
 
-    git clone https://github.com/smnorris/fwapg
-    cd fwapg
-    make --debug=basic
+Connect to the db from your host OS via the port specified in `docker-compose.yml`:
 
-Load/run `bcfishobs`:
+    psql -p 8001 -U postgres bcfishpass
 
-    git clone [email protected]:smnorris/bcfishobs.git
-    cd bcfishobs
-    make --debug=basic
+Stop the containers (without deleting):
 
-Create db schema:
+    docker compose stop
 
-    jobs/db_setup
+Delete the containers:
 
-Load source data:
+    docker compose down
 
-    jobs/load_static                     
-    jobs/load_monthly
-    jobs/load_weekly
-    jobs/load_modelled_stream_crossings
+To build/load/dump a small database for development/testing:
 
-Run the model:
+    docker compose build
+    docker compose up -d
+    cd test
+    docker compose run --rm runner build_db.sh
 
-    jobs/model_stream_crossings    # (optionally - this is only needs to be run on the primary provincial bcfishpass database)
-    jobs/model_prep
-    jobs/model_run
+Note that `build_db.sh` dumps all required inputs to a postgresql dump file - once it has been run once, a testing database can be quickly restored from the dump rather than loading from scratch:
 
-# Backups
+    cd ..
+    docker compose down           # shut down the vm
+    rm -rf postgres-data          # remove the testing db data folder
+    docker compose up -d          # restart docker, create_db is automatically run
+    pg_restore -d $DATABASE_URL test/bcfishpass_test.dump  # call restore from local OS
 
-Backup strategies will vary but it can be useful to dump the entire database to file.  
-This appends the date and commit tag date to the file name:
+Run the models on the testing watershed groups:
 
-        pg_dump -Fc $DATABASE_URL > bcfishpass.$(git describe --tags --abbrev=0).$(date +%F).dump    
+    docker compose run --rm runner test.sh

docker-compose.yaml

@@ -0,0 +1,30 @@
+services:
+  db:
+    build:
+      context: docker/db
+    container_name: bcfishpass-db
+    command: postgres -c default_statistics_target=100 -c max_connections=20 -c max_locks_per_transaction=64 -c checkpoint_timeout=30min -c maintenance_work_mem=1GB -c effective_cache_size=16GB -c work_mem=500MB -c max_wal_size=10GB -c wal_buffers=16MB -c shared_buffers=8GB
+    ports:
+      - ${DB_PORT}:5432
+    shm_size: 16gb
+    volumes:
+      - ./postgres-data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      interval: 5s
+      timeout: 1s
+      retries: 5
+    environment:
+      - POSTGRES_USER=postgres
+      - POSTGRES_PASSWORD=postgres
+      - POSTGRES_DB=bcfishpass_test
+  runner:
+    build: docker/runner
+    container_name: bcfishpass-runner
+    volumes:
+      - "./:/home/bcfishpass"
+    environment:
+      - DATABASE_URL=postgresql://postgres:postgres@db:5432/bcfishpass_test
+    depends_on:
+      db:
+        condition: service_healthy

docker/db/Dockerfile

@@ -0,0 +1,28 @@
+#
+# NOTE: THIS DOCKERFILE IS GENERATED VIA "make update"! PLEASE DO NOT EDIT IT DIRECTLY.
+#
+
+FROM postgres:16-bullseye
+
+LABEL maintainer="PostGIS Project - https://postgis.net" \
+      org.opencontainers.image.description="PostGIS 3.5.0+dfsg-1.pgdg110+1 spatial database extension with PostgreSQL 16 bullseye" \
+      org.opencontainers.image.source="https://github.com/postgis/docker-postgis"
+
+ENV POSTGIS_MAJOR 3
+ENV POSTGIS_VERSION 3.5.0+dfsg-1.pgdg110+1
+
+RUN apt-get update \
+      && apt-cache showpkg postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR \
+      && apt-get install -y --no-install-recommends \
+           # ca-certificates: for accessing remote raster files;
+           #   fix: https://github.com/postgis/docker-postgis/issues/307
+           ca-certificates \
+           \
+           postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR=$POSTGIS_VERSION \
+           postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts \
+      && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir -p /docker-entrypoint-initdb.d
+COPY ./initdb-postgis.sh /docker-entrypoint-initdb.d/10_postgis.sh
+COPY ./update-postgis.sh /usr/local/bin
+

docker/db/initdb-postgis.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+set -e
+
+# Perform all actions as $POSTGRES_USER
+export PGUSER="$POSTGRES_USER"
+
+# Create the 'template_postgis' template db
+"${psql[@]}" <<- 'EOSQL'
+CREATE DATABASE template_postgis IS_TEMPLATE true;
+EOSQL
+
+# Load PostGIS into both template_database and $POSTGRES_DB
+for DB in template_postgis "$POSTGRES_DB"; do
+	echo "Loading PostGIS extensions into $DB"
+	"${psql[@]}" --dbname="$DB" <<-'EOSQL'
+		CREATE EXTENSION IF NOT EXISTS postgis;
+		--CREATE EXTENSION IF NOT EXISTS postgis_topology;
+		-- Reconnect to update pg_setting.resetval
+		-- See https://github.com/postgis/docker-postgis/issues/288
+		--\c
+		--CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;
+		--CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder;
+EOSQL
+done

docker/db/update-postgis.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+
+set -e
+
+# Perform all actions as $POSTGRES_USER
+export PGUSER="$POSTGRES_USER"
+
+POSTGIS_VERSION="${POSTGIS_VERSION%%+*}"
+
+# Load PostGIS into both template_database and $POSTGRES_DB
+for DB in template_postgis "$POSTGRES_DB" "${@}"; do
+    echo "Updating PostGIS extensions '$DB' to $POSTGIS_VERSION"
+    psql --dbname="$DB" -c "
+        -- Upgrade PostGIS (includes raster)
+        CREATE EXTENSION IF NOT EXISTS postgis VERSION '$POSTGIS_VERSION';
+        ALTER EXTENSION postgis  UPDATE TO '$POSTGIS_VERSION';
+
+        -- Upgrade Topology
+        CREATE EXTENSION IF NOT EXISTS postgis_topology VERSION '$POSTGIS_VERSION';
+        ALTER EXTENSION postgis_topology UPDATE TO '$POSTGIS_VERSION';
+
+        -- Install Tiger dependencies in case not already installed
+        CREATE EXTENSION IF NOT EXISTS fuzzystrmatch;
+        -- Upgrade US Tiger Geocoder
+        CREATE EXTENSION IF NOT EXISTS postgis_tiger_geocoder VERSION '$POSTGIS_VERSION';
+        ALTER EXTENSION postgis_tiger_geocoder UPDATE TO '$POSTGIS_VERSION';
+    "
+done

Dockerfile → docker/runner/Dockerfile

@@ -1,5 +1,6 @@
 FROM ghcr.io/osgeo/gdal:ubuntu-full-3.10.0
 
+
 RUN apt-get update && apt-get --assume-yes upgrade \
     && apt-get -qq install -y --no-install-recommends postgresql-common \
     && apt-get -qq install -y --no-install-recommends yes \

jobs/load_monthly

@@ -38,6 +38,47 @@ ogr2ogr -f PostgreSQL \
   /vsicurl/https://nrs.objectstore.gov.bc.ca/bchamp/bcdata/whse_basemapping.transport_line.parquet \
   whse_basemapping.transport_line
 
+#-------
+# refresh code tables - unlikely to change, but simple/quick to load
+#-------
+ogr2ogr \
+  -f PostgreSQL \
+  "PG:$DATABASE_URL" \
+  -append \
+  --config OGR_TRUNCATE YES \
+  -nln whse_basemapping.transport_line_type_code \
+  /vsizip//vsicurl/https://nrs.objectstore.gov.bc.ca/bchamp/bcdata/transport_line.gpkg.zip \
+  transport_line_type_code
+
+ogr2ogr \
+  -f PostgreSQL \
+  "PG:$DATABASE_URL" \
+  -append \
+  --config OGR_TRUNCATE YES \
+  -nln whse_basemapping.transport_line_surface_code \
+  /vsizip//vsicurl/https://nrs.objectstore.gov.bc.ca/bchamp/bcdata/transport_line.gpkg.zip \
+  transport_line_surface_code
+
+ogr2ogr \
+  -f PostgreSQL \
+  "PG:$DATABASE_URL" \
+  -append \
+  --config OGR_TRUNCATE YES \
+  -nln whse_basemapping.transport_line_divided_code \
+  /vsizip//vsicurl/https://nrs.objectstore.gov.bc.ca/bchamp/bcdata/transport_line.gpkg.zip \
+  transport_line_divided_code
+
+ogr2ogr \
+  -f PostgreSQL \
+  "PG:$DATABASE_URL" \
+  -append \
+  --config OGR_TRUNCATE YES \
+  -nln whse_basemapping.transport_line_structure_code \
+  /vsizip//vsicurl/https://nrs.objectstore.gov.bc.ca/bchamp/bcdata/transport_line.gpkg.zip \
+  transport_line_structure_code
+
+
+
 # parcel fabric, renaming download so it can be read without unzipping
 # ----
 curl \

jobs/replicate_dra

@@ -17,7 +17,7 @@ unzip \
   -o \
   /tmp/transport_line.gdb.zip
 
-# post public portion of dataset to bchamp object storage
+# write required layer and lookups
 ogr2ogr \
   -f Parquet \
   /tmp/transport_line.parquet \

model/03_habitat_lateral/valley_confinement.py

@@ -286,7 +286,7 @@ def get_precip(db, bounds, DEM, dem_meta, data_path):
     A = features.rasterize(
         (
             (geom, value)
-            for geom, value in zip(precip_features.geometry, precip_features.map)
+            for geom, value in zip(precip_features.geometry, precip_features["map"])
         ),
         out_shape=DEM.shape,
         transform=dem_meta["transform"],

test/README.md

@@ -1,14 +1,14 @@
 # A minimal db for development and testing
 
-## Create dump file
+## Create database dump file
 
-Bootstrap database `bcfishpass_test` with fwapg/bcfishobs, load selected data, dump to file, drop db:
+Bootstrap database `bcfishpass_test` with latest schema, fwapg/bcfishobs, load selected data, dump to file:
 
-    ./build.sh
+    ./build_db.sh
 
-## Testing usage
+## Test bcfishpass scripts
 
-Restore fwapg/bcfishobs db from dump, load data, run model:
+Run model:
 
     ./test.sh