Add initial SpatiaLite ↔ PostGIS support (#96)

Co-authored-by: mkgrgis <[email protected]>

.github/workflows/CI.yml

@@ -61,9 +61,9 @@ jobs:
       - name: install SQLite, ${{ matrix.config }} mode
         run: |
           if [[ "${{ matrix.config }}" == "default" ]]; then
-            bash GitHubActions/install_sqlite.sh ${{ env.SQLITE_VERSION }} ${{ env.SQLITE_YEAR }}
+            bash GitHubActions/install_sqlite.sh ${{ env.SQLITE_VERSION }} ${{ env.SQLITE_YEAR }} "${{ matrix.config }}"
           elif [[ "${{ matrix.config }}" == "postgis" ]]; then
-            bash GitHubActions/install_sqlite.sh ${{ env.SQLITE_VERSION }} ${{ env.SQLITE_YEAR }} --enable-rtree
+            bash GitHubActions/install_sqlite.sh ${{ env.SQLITE_VERSION }} ${{ env.SQLITE_YEAR }} "${{ matrix.config }}" --enable-rtree
           fi
 
       - name: build sqlite_fdw, ${{ matrix.config }} mode

GIS.md

@@ -0,0 +1,98 @@
+GIS support in SQLite Foreign Data Wrapper for PostgreSQL
+=========================================================
+
+<img src="https://www.tmapy.cz/wp-content/uploads/2021/02/postgis-logo.png" align="center" height="80" alt="PostGIS"/>	+ <img src="https://www.gaia-gis.it/fossil/libspatialite/logo" align="center" height="80" alt="SpatiaLite"/>
+
+SQLite FDW for PostgreSQL can connect PostgreSQL with or without [PostGIS](https://www.postgis.net/)
+to [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) SQLite database file.
+This description contains only information about GIS support without common SQL and
+system descriptions from [common FDW description](README.md).
+
+Notes about compilation environment and PROJ library
+----------------------------------------------------
+Both SpatiaLite and PostGIS uses [PROJ C++ library](https://github.com/OSGeo/PROJ) from
+[Open Source Geospatial Foundation](https://github.com/OSGeo) for reprojecting and some calculations.
+Recommended configuration of SQLite FDW with GIS support contains both SpatiaLite and PostGIS.
+**It is recommended to install only 1 version of PROJ in a system to avoid integration issue**.
+Before compilation you should ensure equal versions of PROJ library required by SpatiaLite and PostGIS on
+your system. Otherwise some encapsualted and unwanted memory freeing errors can occur.
+If you need different PROJ library versions, you can reference [this PROJ issue](https://github.com/OSGeo/PROJ/issues/4361)
+and try to use something like `CFLAGS="$CFLAGS -DPROJ_RENAME_SYMBOLS -O2"` during compilation of
+SpatiaLite or PostGIS to link one of this extensions with other PROJ version.
+
+Common conditions of GIS support
+--------------------------------
+
+1. SQLite FDW should be compiled with `ENABLE_GIS=1` environment variable value.
+2. You must install SpatiaLite header files before compilation.
+Linux packages like `libspatialite-dev` or `libspatialite-devel` can contain this files.
+3. A column should have data type (domain) name from following list:
+	* addbandarg
+	* box2d
+	* box3d
+	* geography
+	* geometry
+	* geometry_dump
+	* geomval
+	* getfaceedges_returntype
+	* rastbandarg
+	* raster
+	* reclassarg
+	* summarystats
+	* topoelement
+	* topoelementarray
+	* topogeometry
+	* unionarg
+	* validatetopology_returntype
+
+	Only listed data types have full data transformation support:
+	* geography
+	* geometry
+
+All other data types (domains) are treated as PostGIS specific, but unsupported.
+
+You can use SpatiaLite GIS data support _without PostGIS installation_ after such
+SQL commands as `CREATE DOMAIN geometry AS bytea;` and `CREATE DOMAIN geography AS bytea;`.
+This allows to have in PostgreSQL PostGIS compatible `bytea` data easy
+convertable to PostGIS storage format.
+
+PostGIS and SpatiaLite vector data formats
+-------------------------------------------
+
+Vector GIS data in PostGIS can be stored in a columns with `geography` or `geometry`
+data type. This columns contains a binary data.
+[Well-known binary (WKB)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary)
+data storage format is a standard of [Open Geospatial Consortium (OGC)](https://en.wikipedia.org/wiki/Open_Geospatial_Consortium)
+and supported by [GEOS library](https://libgeos.org). PostGIS internal GIS data
+storage format based on WKB with [SRID](https://en.wikipedia.org/wiki/Spatial_reference_system#Identifiers)
+additions. This format is known as [EWKB](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Format_variations) and supported by
+[GEOS library](https://libgeos.org) and SpatiaLite input/output functions.
+
+Hexadecimal text representation of EWKB data is a transport form for `geography`
+and `geometry` GIS data between PostgreSQL and *SpatiLite input/output functions*.
+Hence no PostGIS input/output functions are necessary, but all of this functions
+are supported.
+
+EWKB hexadecimal text data transport is faster than
+[EWKT](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry)
+but slower than EWKB blob data transport.
+
+SpatiaLite internal storage based on `blob` data [affinity](https://www.sqlite.org/datatype3.html)
+and is not a standard of [OGC](https://en.wikipedia.org/wiki/Open_Geospatial_Consortium).
+Also this format doesn't supported by [GEOS library](https://libgeos.org).
+
+Limitations
+-----------
+
+* In opposition to PostGIS, **SpatiaLite doesn't allow to store any GIS vector data without SRID**.
+Hence any well-formed SpatiaLite data can be converted for PostGIS, but
+well-formed PostGIS data _without SRID_ cannot be converted for SpatiaLite.
+All of SpatiaLite input functions will return `NULL` in this case.
+Please use [ST_SetSRID PostGIS function](https://postgis.net/docs/ST_SetSRID.html)
+in case of incomplete SRID data to prepare PostGIS data for importing to SpatiaLite
+or comparing with SpatiaLite data.
+
+* Only `=` PostgreSQL operator is pushed down to SQLite (SpatiaLite) for vector GIS data such
+as `geography` or `geometry`. `<>` PostgreSQL operator is NOT pushed down.
+
+End of description.

GitHubActions/build_postgis.sh

@@ -5,12 +5,12 @@
 # This script downloads PostgreSQL from the official web site into ./workdir
 # then builds it.
 #
-# Usage: ./build_postgres.sh pg_version postgis_version
+# Usage: ./build_postgis.sh pg_version postgis_version
 #     pg_version is a PostgreSQL version to be installed like 16.0.
 #     postgis_version is a PostGIS version to be installed.
 #
 # Requirements
-# - be able to connect to the PostgreSQL official web site by curl.
+# - be able to connect to the PostGIS official web site by wget.
 #
 ################################################################################
 

GitHubActions/build_postgres.sh

@@ -14,7 +14,7 @@
 #
 ################################################################################
 
-VERSION=$1
+POSTGRESQL_VERSION=$1
 CONFIGURE_OPTIONS=""
 
 while (( "$#" )); do
@@ -24,9 +24,9 @@ done
 
 mkdir -p ./workdir
 cd ./workdir
-curl -O https://ftp.postgresql.org/pub/source/v${VERSION}/postgresql-${VERSION}.tar.bz2
-tar xjf postgresql-${VERSION}.tar.bz2
-cd postgresql-${VERSION}
+curl -O https://ftp.postgresql.org/pub/source/v${POSTGRESQL_VERSION}/postgresql-${POSTGRESQL_VERSION}.tar.bz2
+tar xjf postgresql-${POSTGRESQL_VERSION}.tar.bz2
+cd postgresql-${POSTGRESQL_VERSION}
 
 if [ -z "$CONFIGURE_OPTIONS" ]; then
   ./configure

GitHubActions/build_sqlite_fdw.sh

@@ -14,7 +14,7 @@
 # - SQLite development package is installed in a system.
 ################################################################################
 
-VERSION=$1
+VERSION="$1"
 MODE="$2"
 
 mkdir -p ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw

GitHubActions/execute_test.sh

@@ -26,8 +26,7 @@ MODE="$2"
 cd ./workdir/postgresql-${VERSION}/contrib/sqlite_fdw
 
 if [ "$MODE" == "postgis" ]; then
-	MAKEFILE_OPT="ENABLE_GIS=1"
-	echo "$MODE mode, makefile option = $MAKEFILE_OPT"
+	export ENABLE_GIS=1
 
 	# Start postgres server
 	POSTGRES_HOME=/usr/local/pgsql

GitHubActions/install_sqlite.sh

@@ -5,26 +5,28 @@
 # This sript downloads SQLite source code from the official web site into
 # ./workdir then builds and installs it.
 #
-# Usage: ./install_sqlite.sh version year [configure_options]
-#     version: SQLite version to be installed
+# Usage: ./install_sqlite.sh version year testing_mode [configure_options]
+#     version: SQLite version to be installed.
 #     year: A year of SQLite released. It is used for determining a download URL.
-#     configure_options are a list of option for sqlite server.
+#     testing_mode:	'default' or 'postgis' value.
+#     configure_options: are a list of option for sqlite server.
 #
-#     Ex) ./install_sqlite.sh 3420000 2023 --enable-rtree
+#     Ex) ./install_sqlite.sh 3420000 2023 postgis --enable-rtree
 #
 # Requirements
 # - be able to connect to the SQLite official web site by curl.
 # - having superuser privileges
 #
 ################################################################################
 
-VERSION=$1
-YEAR=$2
+VERSION="$1"
+YEAR="$2"
+TESTING_MODE="$3"
 
 CONFIGURE_OPTIONS=""
 
 while (( "$#" )); do
-  CONFIGURE_OPTIONS="$CONFIGURE_OPTIONS $3"
+  CONFIGURE_OPTIONS="$CONFIGURE_OPTIONS $4"
   shift
 done
 
@@ -42,3 +44,7 @@ fi
 
 make
 sudo make install
+
+if [ "$TESTING_MODE" == "postgis" ]; then
+  sudo apt-get install libspatialite-dev -y
+fi

Makefile

@@ -10,13 +10,20 @@
 ##########################################################################
 
 MODULE_big = sqlite_fdw
-OBJS = connection.o option.o deparse.o sqlite_query.o sqlite_fdw.o sqlite_data_norm.o
+OBJS = connection.o option.o deparse.o sqlite_query.o sqlite_fdw.o sqlite_data_norm.o sqlite_gis.o
 
 EXTENSION = sqlite_fdw
 DATA = sqlite_fdw--1.0.sql sqlite_fdw--1.0--1.1.sql
 
+ifdef ENABLE_GIS
+override PG_CFLAGS += -DSQLITE_FDW_GIS_ENABLE
+GISTEST=postgis
+else
+GISTEST=nogis
+endif
+
 ifndef REGRESS
-REGRESS = extra/sqlite_fdw_post types/bitstring types/bool types/float4 types/float8 types/int4 types/int8 types/numeric types/macaddr types/macaddr8 types/out_of_range types/timestamp types/uuid extra/join extra/limit extra/aggregates extra/prepare extra/select_having extra/select extra/insert extra/update extra/encodings sqlite_fdw type aggregate selectfunc 
+REGRESS = extra/sqlite_fdw_post types/bitstring types/bool types/float4 types/float8 types/int4 types/int8 types/numeric types/$(GISTEST) types/macaddr types/macaddr8 types/out_of_range types/timestamp types/uuid extra/join extra/limit extra/aggregates extra/prepare extra/select_having extra/select extra/insert extra/update extra/encodings sqlite_fdw type_$(GISTEST) aggregate selectfunc 
 endif
 
 REGRESS_OPTS = --encoding=utf8
@@ -33,6 +40,10 @@ endif
 
 SHLIB_LINK := -lsqlite3
 
+ifdef ENABLE_GIS
+override SHLIB_LINK += -lspatialite
+endif
+
 ifdef USE_PGXS
 PG_CONFIG = pg_config
 PGXS := $(shell $(PG_CONFIG) --pgxs)
@@ -43,7 +54,6 @@ endif
 ifeq (,$(findstring $(MAJORVERSION), 13 14 15 16 17))
 $(error PostgreSQL 13, 14, 15, 16 or 17 is required to compile this extension)
 endif
-
 else
 subdir = contrib/sqlite_fdw
 top_builddir = ../..
@@ -60,3 +70,9 @@ endif
 REGRESS := $(addprefix $(REGRESS_PREFIX_SUB)/,$(REGRESS))
 $(shell mkdir -p results/$(REGRESS_PREFIX_SUB)/extra)
 $(shell mkdir -p results/$(REGRESS_PREFIX_SUB)/types)
+
+ifdef ENABLE_GIS
+check: temp-install
+temp-install: EXTRA_INSTALL+=contrib/postgis
+checkprep: EXTRA_INSTALL+=contrib/postgis
+endif