Document format x to GML conversion options

[stijnvanhoey]

With the addition of the location query options #129 we support the usage of GML to query on location. However, users will probably have other file formats (shapefile, geojson,...) to use within the location search context.

Without supporting more formats within the pydov package, we could point the user to some other references or code examples on how to deal with other file formats.

@jorisvandenbossche, taking into account your experiences on spatial matter, do you have pointers or ideas that we can include in the documentation on how to convert xxxx to GML? with xxxx equal to shapefile, geojson,... (feature file types)

[jorisvandenbossche]

As far as I know, it is not that straightforward to convert shapefiles, geojson, .. to GML with python. At least, fiona does not support it. A user could first convert it with GDAL command line if his/her GDAL is built with gml support.

But, I think there would be some ways to provide some useful functionality, assuming the user has a GeoDataFrame (so a shapefile, geojson, ... read with fiona/geopandas).

• To start with, a GeoDataFrame has a total_bounds attribute (https://geopandas.readthedocs.io/en/latest/reference.html#geopandas.GeoSeries.total_bounds), which gives you back the (minx, miny, maxx, maxy) tuple.
  So I think (didn't test) one could already do: Box(*gdf.total_bounds) where gdf is the GeoDataFrame. For this you could certainly add a documentation example.

• Another idea would be to add support for __geo_interface__ to pydov (see https://gist.github.com/sgillies/2217756 for the interface standard).

  • You could eg check for that attribute for any passed object to Box, and then get the bbox entry of the dict, so this would enable a user to do: Box(gdf) instead of Box(*gdf.total_bounds) (although the total_bounds makes it more explicit that this is what is happening ..).
  • This functionality could also be added to functions that expect a Point (a shapely Point has also this __geo_interface__, eg check from shapely.geometry import Point; Point(1, 1).__geo_interface__).
    That would make interaction a bit more fluent, as you could do something like WithinDistance(gdf.unary_union.centroid, distance=100) (where gdf.unary_union.centroid is a shapely Point calculated as the centroid of the full shapefile).

• But to use directly the shapes (eg a set of polygons) of a shapefile, geojson, ..., the user will either need to convert it to GML, or you would need to add something like pydov.util.location.Polygon in a similar way as you have Point, but of course then you would need to add generation of the gml for a polygon, which will be more complicated as for Point.

---

There is actually a rather easy way to create small snippets of gml, and that is by using GDAL/OGR directly:

In [60]: import ogr

In [61]: geojson = """{"type":"Point","coordinates":[108420.33,753808.59]}"""

In [62]: point = ogr.CreateGeometryFromJson(geojson) 

In [63]: point.ExportToGML()
Out[63]: '<gml:Point srsName="EPSG:4326"><gml:coordinates>108420.33,753808.59</gml:coordinates></gml:Point>'

(and where the geojson string could be easily retrieved from the __geo_interface__)

Only, it is apparently not really recommended to combine usage of ogr directly and fiona (which already used by GeoPandas, in case you would use that to read the shapefile, geojson file, ...): granularag/pyspatial#22, https://rasterio.readthedocs.io/en/latest/topics/switch.html#mutual-incompatibilities

---

BTW, this looks like really nice functionality. Is it something that would eventually maybe fit in OWSLib ? (like you now use the filter expressions system already from OWSLib)

[pjhaest]

Thanks for the explanation! Do I get it right that you should not combine ogr with geopandas? Or that you should take care not to mix objects between both?

[pjhaest]

Discussed on code spring May 2019: do not mix imports of rasterio with gdal/fiona/geopandas as indicated in the references above

[jorisvandenbossche]

do not mix imports of rasterio with gdal/fiona/geopandas as indicated in the references above

Small correction: I think it needs ot be "do not mix imports of rasterio/fiona/geopandas with gdal". It's only gdal's own python bindings that don't work with the ecosystem of geo packages around rasterio, fiona, etc

[jorisvandenbossche]

Just thought about this, there is actually a way to let fiona write GML, by explicitly saying to fiona that it is OK to do so.
Then that works, but if it creates proper GML, I don't know (no expert in that). Small example:

In [16]: import geopandas                                                                                                                                                                                          

In [17]: import fiona                                                                                                                                                                                              

In [18]: df = geopandas.read_file(geopandas.datasets.get_path('naturalearth_cities'))                                                                                                                              

In [19]: df.head()                                                                                                                                                                                                 
Out[19]: 
           name                                     geometry
0  Vatican City  POINT (12.45338654497177 41.90328217996012)
1    San Marino    POINT (12.44177015780014 43.936095834768)
2         Vaduz  POINT (9.516669472907267 47.13372377429357)
3    Luxembourg  POINT (6.130002806227083 49.61166037912108)
4       Palikir  POINT (158.1499743237623 6.916643696007725)

In [20]: fiona.supported_drivers['GML'] = 'rw'                                                                                                                                                                     

In [21]: df.to_file("test.gml", driver='GML')    

gives

$ !head test.gml -n 25                                                                                                                                                                                      
<?xml version="1.0" encoding="utf-8" ?>
<ogr:FeatureCollection
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://ogr.maptools.org/ test.xsd"
     xmlns:ogr="http://ogr.maptools.org/"
     xmlns:gml="http://www.opengis.net/gml">
  <gml:boundedBy>
    <gml:Box>
      <gml:coord><gml:X>-175.2205644776166</gml:X><gml:Y>-41.29997393927641</gml:Y></gml:coord>
      <gml:coord><gml:X>179.2166470940289</gml:X><gml:Y>64.15002361973922</gml:Y></gml:coord>
    </gml:Box>
  </gml:boundedBy>
                                                                                              
  <gml:featureMember>
    <ogr:test fid="test.0">
      <ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>12.4533865449718,41.9032821799601</gml:coordinates></gml:Point></ogr:geometryProperty>
      <ogr:name>Vatican City</ogr:name>
    </ogr:test>
  </gml:featureMember>
  <gml:featureMember>
    <ogr:test fid="test.1">
      <ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>12.4417701578001,43.936095834768</gml:coordinates></gml:Point></ogr:geometryProperty>
      <ogr:name>San Marino</ogr:name>
    </ogr:test>
  </gml:featureMember>

[rebot]

Hi,

I've made a conversion from Shapefile to gml that can be used to filter pyDOV.
I make use of GeoPandas, Fiona and Shapely. I've installed Fiona and GDAL using the wheel provided on the following website.

GDAL Version 3.1.3

Fiona @ file:///C:/Users/BEGILT/Development/Python/RockWorks/Fiona-1.8.17-cp36-cp36m-win_amd64.whl
GDAL @ file:///C:/Users/BEGILT/Development/Python/RockWorks/GDAL-3.1.3-cp36-cp36m-win_amd64.whl

Workflow

# In[1]: Laad de projectzone is - projectzone bestaat uit verschillende polygonen

segmenten = gpd.read_file(os.path.join('hulpmiddelen','projectzone','segmenten.shp'))

# In[2]: Filter de geometrie en zet om naar een lijst punten

# Schema = http://schemas.opengis.net/gml/ - Schema definitie voor de .gml
schema = {'properties': OrderedDict([]),'geometry': 'Polygon'}

# https://gis.stackexchange.com/questions/52705/how-to-write-shapely-geometries-to-shapefiles
# https://geoscripting-wur.github.io/PythonVector/

def exporteer_vorm(segment):
    # Haal GML op schrijf de file weg in memory
    gml_bytes = BytesIO()
    gml_filename = os.path.join('hulpmiddelen','projectzone',f'{segment.segment}.gml')
    with fiona.open(gml_bytes, 'w', driver='GML', schema=schema, crs=from_epsg(31370)) as f:
        f.write({
            'geometry': mapping(segment.geometry),
            'properties': {}
        })
    gml_bytes.seek(0)
    # Omzetting .gml van versie 2 naar versie 3.1.1 dat bruikbaar is in pyDOV
    gml_content = gml_bytes.read().decode('utf-8')
    gml_content = gml_content.replace('Box','Envelope')
    gml_content = gml_content.replace('coord><gml:X', 'lowerCorner', 1)
    gml_content = gml_content.replace('Y></gml:coord', 'lowerCorner', 1)
    gml_content = gml_content.replace('coord><gml:X', 'upperCorner', 1)
    gml_content = gml_content.replace('Y></gml:coord', 'upperCorner', 1)
    gml_content = gml_content.replace('</gml:X><gml:Y>', ' ')
    gml_content = gml_content.replace('fid', 'gml:id')
    gml_content = gml_content.replace('outerBoundaryIs', 'exterior')
    gml_content = gml_content.replace('coordinates', 'posList')
    gml_content = gml_content.replace(',', ' ')
    gml_content,_ = re.subn(r'\n\n', r'\n', gml_content)
    gml_id = re.search('ogr:(\w{32})\s',gml_content).group(1)
    gml_content = gml_content.replace(gml_id, segment.segment)
    # Schrijf het omgezette bestand weg naar een .gml
    with open(gml_filename, 'w', newline='') as f:
        f.write(gml_content)  

# Voer de functie uit voor elke rij uit de DataFrame
segmenten.apply(exporteer_vorm, axis=1)

Problem

I have to convert the .gml from version 2 to version 3.1.1 using some additional lines of code. You can add some additional keyword in the fiona.open function, but it didn't worked out. I like to be able to save my file as a 3.1.1 .gml using Fiona.

Something like this (see the XSD), but It didn't worked out :

fiona.open(gml_bytes, 'w', driver='GML', schema=schema, crs=from_epsg(31370), XSD='http://schemas.opengis.net/gml/3.1.1/base/feature.xsd')

Can somebody help me with this? Afterwards, I'll fork pyDOV and add an example.

[pjhaest]

sorry, no experience yet, maybe ask at fiona first?

[rebot]

There seems to be a bug in Fiona, an issue has been made to fix this! Once the issue has been resolved, I'll provide pydov with an example. EDIT: Fix has been implemented and is scheduled to be included in the next version of Fiona that will be released in November

[Roel]

Yes, an example for using a Shapefile as a spatial filter in pydov would be very interesting. SHP is (unfortunately) still widely used, so adding this to the documentation would benefit a lot of users.

Maybe we can enlarge the scope of this tutorial notebook to "spatial querying" and include it there? We can then refer to that notebook in the main documentation on spatial querying.

[rebot]

After #309 get merged, this issue can be closed.