Datapackage Support for geonode-importer

This contrib app provides a datapackage handler for the geonode-importer. The app aims to be compatible with geonode-importer v1.0.9, but can be used only with GeoNode v4.3.x.

The handler leverages the OGR VRT format to import non-spatial datasets described in a tabular data resource schema. To make this possible a schema mapping takes place.

🔧 Development Hint

There are lots of adjustments made in both GeoNode, geonode-mapstore-client which are necessary to integrate tabular data. You should be able to the see all needed changes by comparing branches:

• 🔄 Compare GeoNode changes
• 🔄 Compare geonode-mapstore-client changes (you can ignore all changes under /geonode_mapstore_client/client/static/mapstore/dist of course)

Check out the development setup (based on the Geonode Docker blueprint).

Feel free to contribute by creating an issue or even a PR on GitHub.

Installation

The following steps are necessary so GeoNode is able to handle the frictionless datapackage format:

• Installation of the datapackage handler for the geonode-importer
• Manual integration of a UI code adjustments
• Configuration of the MapStore plugin

Install Datapackage Handler

The module can be installed as normal Django contrib app:

INSTALLED_APPS += (
   importer_datapackage
)

Install all necessary dependencies via pip install -r requirements.txt. For development let pip install the branch of your current interest:

-e git+https://github.com/GeoNodeUserGroup-DE/importer-datapackage.git@main#egg=importer_datapackage

The URL above references the main branch.

Once installed, the importer_datapackage.handlers.datapackage.handler.DataPackageFileHandler has to be added to the IMPORTER_HANDLERS. Open the settings.py and add

IMPORTER_HANDLERS = (
    'importer_datapackage.handlers.datapackage.handler.DataPackageFileHandler',
    *IMPORTER_HANDLERS,
)

Geonode Mapstore Client Plugin

There is a geonode-mapstore-client plugin to preview tabular data. To let the UI recognize and handle tabular datasets, you have to integrate code which resides in branch datapackage_tabular-data of the GeoNodeUserGroup-DE geonode-mapstore-client fork.

Configuration

To enable the geonode-mapstore-plugin add the following plugin configuration to your localConfig.json:

"plugins": {
    
   "tabular_viewer":[
      {
         "name":"ActionNavbar",
         "cfg":{
            "containerPosition":"header",
            "titleItems":[
               {
                  "type":"plugin",
                  "name":"DetailViewerButton"
               }
            ],
            "leftMenuItems":[
               {
                  "labelId":"gnviewer.edit",
                  "type":"dropdown",
                  "disableIf":"{!context.resourceHasPermission(state('gnResourceData'), 'change_resourcebase')}",
                  "items":[
                     {
                        "type":"plugin",
                        "name":"DetailViewerButton"
                     },
                     {
                        "type":"link",
                        "href":"{'#/dataset/' + (state('gnResourceData') || {}).pk + '/edit/data'}",
                        "labelId":"gnviewer.editData",
                        "disableIf":"{!context.resourceHasPermission(state('gnResourceData'), 'change_dataset_data')}"
                     },
                     {
                        "type":"link",
                        "href":"{'#/dataset/' + (state('gnResourceData') || {}).pk + '/edit/style'}",
                        "labelId":"gnviewer.editStyle",
                        "disableIf":"{!context.resourceHasPermission(state('gnResourceData'), 'change_dataset_style')}"
                     },
                     {
                        "type":"link",
                        "href":"{context.getMetadataUrl(state('gnResourceData'))}",
                        "labelId":"gnviewer.editMetadata"
                     },
                     {
                        "type":"link",
                        "href":"{'/datasets/' + (state('gnResourceData') || {}).alternate + '/style_upload'}",
                        "labelId":"gnviewer.styleUpload",
                        "disableIf":"{!context.resourceHasPermission(state('gnResourceData'), 'change_dataset_style')}"
                     },
                     {
                        "type":"link",
                        "href":"{'/datasets/' + (state('gnResourceData') || {}).alternate + '/metadata_upload'}",
                        "labelId":"gnviewer.metadataUpload"
                     }
                  ]
               },
               {
                  "labelId":"gnviewer.view",
                  "type":"dropdown",
                  "disableIf":"{context.resourceHasPermission(state('gnResourceData'), 'change_resourcebase')}",
                  "items":[
                     {
                        "type":"link",
                        "href":"{context.getMetadataDetailUrl(state('gnResourceData'))}",
                        "labelId":"gnviewer.viewMetadata"
                     },
                     {
                        "type":"link",
                        "href":"{'#/dataset/' + (state('gnResourceData') || {}).pk + '/edit/data'}",
                        "labelId":"gnviewer.viewData",
                        "disableIf":"{state('gnResourceData') && (state('gnResourceData').subtype === 'raster' || state('gnResourceData').subtype === 'remote')}"
                     }
                  ]
               },
               {
                  "type":"plugin",
                  "name":"Share"
               },
               {
                  "type":"plugin",
                  "name":"DeleteResource",
                  "disableIf":"{!context.resourceHasPermission(state('gnResourceData'), 'delete_resourcebase')}"
               },
               {
                  "type":"divider",
                  "authenticated":true
               },
               {
                  "type":"plugin",
                  "name":"Print"
               },
               {
                  "labelId":"gnviewer.download",
                  "disablePluginIf":"{!state('selectedLayerPermissions').includes('download_resourcebase')}",
                  "type":"dropdown",
                  "items":[
                     {
                        "type":"plugin",
                        "name":"LayerDownload"
                     },
                     {
                        "type":"plugin",
                        "name":"IsoDownload"
                     },
                     {
                        "type":"plugin",
                        "name":"DublinCoreDownload"
                     }
                  ]
               }
            ],
            "rightMenuItems":[
               {
                  "type":"plugin",
                  "name":"FullScreen"
               }
            ]
         }
      },
      {
         "name":"Share",
         "cfg":{
            "containerPosition":"rightOverlay",
            "enableGeoLimits":true
         }
      },
      {
         "name":"DetailViewer",
         "cfg":{
            "containerPosition":"rightOverlay"
         }
      },
      {
         "name":"DeleteResource"
      },
      {
         "name":"FullScreen"
      },
      {
         "name":"IsoDownload"
      },
      {
         "name":"DublinCoreDownload"
      },
      {
         "name": "LayerDownload",
         "cfg": {
            "disablePluginIf": "{!state('selectedLayerPermissions').includes('download_resourcebase')}",
            "defaultSelectedService": "wfs",
            "hideServiceSelector": true,
            "formats": [
               {"name": "excel", "label": "excel", "validServices": ["wfs"]},
               {"name": "csv", "label": "CSV", "validServices": ["wfs"]}
            ]
         }
      },
      {
         "name":"FullScreen",
         "override":{
            "Toolbar":{
               "alwaysVisible":true
            }
         }
      },
      {
         "name":"Notifications"
      },
      {
         "name":"TabularPreview"
      }
   ]
}

Upload

A datapackage can be uploaded as ZIP-file. In the ZIP-file a datapackage.json is expected to be found, along with any other file referenced from the datapackage.json. At the moment only CSV-files were tested.

You can use this test zip file to start.

Limitations

• no fancy formats or regexes
• geonode-mapstore-client currently mixes file specs for upload
• primary keys are imported without any database constraints
• ...

Also check the issues page for more quirks and findings.

Funding

This contrib app was funded by

Logo	Funder
	Thünen-Institute
	Leibniz Centre for Agricultural Landscape Research