Skip to content

Commit

Permalink
First version
Browse files Browse the repository at this point in the history
  • Loading branch information
hbruch committed Mar 16, 2020
1 parent ef531c1 commit e4f7b08
Show file tree
Hide file tree
Showing 16 changed files with 1,207 additions and 0 deletions.
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data/
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data/
27 changes: 27 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
FROM alpine
MAINTAINER MITFAHR|DE|ZENTRALE <[email protected]>

RUN apk add --update --no-cache \
make \
bash \
curl \
zip \
&& rm -rf /var/cache/apk/*

WORKDIR /opt/gtfs-hub

RUN curl -O https://download.docker.com/linux/static/stable/x86_64/docker-18.06.1-ce.tgz && \
tar xzf docker-18.06.1-ce.tgz && \
cp docker/docker /usr/bin/docker && \
rm -rf docker*

VOLUME /var/data

ADD update_gtfs.sh .
ADD update_osm.sh .
ADD update_all.sh .
ADD makefile .

ADD config/ ./config/

CMD bash ./update_all.sh
77 changes: 77 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# GTFS-Hub

This project aims at providing community tested, probably quality enhanced, partially merged GTFS-feeds of (currently) German transport agencies.

In contrast to a Germany-wide GTFS feed distributed via Delfi (the national access point), GTFS-hub collects and enhances feeds provided by local authorities.

## Motivation
Currently, the national access point provides timetable data in NeTEx format. Via [gtfs.de](http://gtfs.de) derived GTFS publications are available.

However, for upstreaming data to the national data access point seems not to be lossfree or regularly. And there is no feedback channel to fix quality issues in a timely manner

While we hope, that all these problems are overcome soon, we currently still see a need to work with locally published GTFS data.

## Inner workings

### Updating, checking, enhancing GTFS
GTFS-Hub regularly checks on a list of well known GTFS-feeds, if the were updated.

If yes, they are

* downloaded,
* optionally enhanced with shapes using OSM data and the pfaedle tool
* quality checked via google's transitfeed feedvalidator
* optionally transformed with onebusaway transformer tool (fed with a feed specific rule file)
* and optionally merged into larger aggregated GTFS feeds

### Updating and preparing OSM data
Before GFTS data is updated, the OSM data which is used to generate GFTFS shapes is updated.
To avoid daily downloading large pbf datasets (GTFS-Hub downloads DACH (Germany, Austria, Switzerland) and Alsace (France)) from scratch, we only download the original datases once
and update these afterwards via pyosmium and prepares some region clipped extracts (namely Baden-Württemberg including a buffer of some kilometers around the border).

As this extract will serve as input to OpenTripPlanner as well, we do some additionally data processing on it to enhance some infos, e.g.

* Set some parkings to park_ride=yes for parkings close to stations but no park_ride tag yet
* Set some well known parkings to park_ride=hov

### Publishing
After updating OSM and GTFS data, you'll find the datasets in data/www, ready to publish e.g. via a web serve serving this directory.

### External references
This project uses a couple of other dockerized applications:

* mfdz/pfaedle: a dockerized version of Patrick Brosi's tool pfaedle the enhance GTFS feeds by shapes map matched using OSM data
* mfdz/pyosmium: a dockerized version of (py)osmium to quickly update / cut / merge OSM data
* mfdz/osmosis: a dockerized version of osmosis to enhance OSM data with a declarative instruction set
* mfdz/transitfeed: a dockerized version of google's transitfeed feedvalidator
* mfdz/otp-data-tools: a dockerized version of onebusaway's GTFS transform and merge tools

Thanks to everybody contributing to these tools, the OSM community and Geofabrik and the transit agencies providing the data download services.

## How to start gtfs-hub

### Prerequisites

You'll need to have docker installed.

### Running GTFS-Hub
Start the download/transform process chain. Note the necessary HOST_DATA environment variable, which requires to be set to an absolute path to the data directory, as we use a Docker in Docker setup
where data is shared via host-relative volumes.

```sh
docker run -e HOST_DATA=$(PWD)/data -v $(PWD)/data:/var/data -v /var/run/docker.sock:/var/run/docker.sock mfdz/gtfs-hub
```

If you want to use your own config instead, you may mount your own config directory, which

```sh
docker run -e HOST_DATA=$(PWD)/data -v $(PWD)/data:/var/data -v /var/run/docker.sock:/var/run/docker.sock -v $(PWD)/config:/opt/gtfs-hub/config mfdz/gtfs-hub
```

### Building the docker image
To build you own docker image, just do:

```sh
docker build -t mfdz/gtfs-hub .
```

41 changes: 41 additions & 0 deletions config/gtfs-feeds.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
shortname;license;attribution;permanentUrl?;downloadUrl;infoUrl;infoEmail;enhanceShapes
AVV;cc-zero;;Ja;http://opendata.avv.de/current_GTFS/AVV_GTFS_mit_SPNV.zip;https://avv.de/de/fahrplaene/angebote-fuer-webentwickler;;Nein
bodo;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/bodo.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
DING;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/ding.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
Filsland;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/filsland.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
FlixBus;?;?;ja;http://data.ndovloket.nl/flixbus/flixbus-eu.zip;http://data.ndovloket.nl/flixbus/;Nein
HNV;dl-de/by-2.0;Datensatz der NVBW GmbH;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_ohne_Liniennetz/hnv.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-ohne-liniennetz/;[email protected];Ja
HVG;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/hvg.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
HVV;dl-de/by-2.0;Hamburger Verkehrsverbund GmbH;Nein;http://daten.transparenz.hamburg.de/Dataport.HmbTG.ZS.Webservice.GetRessource100/GetRessource100.svc/63302429-4b7d-46c7-aba2-c6e8e090f286/Upload__HVV_Rohdaten_GTFS_Fpl_20200312.zip;http://suche.transparenz.hamburg.de/?q=gtfs;;Nein
KVSH;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/kvsh.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
KVV;cc-zero;;Ja;https://projekte.kvv-efa.de/GTFS/google_transit.zip;https://www.kvv.de/fahrt-planen/fahrplaene/open-data.html;;Ja
LVB;dl-de/by-2.0;Leipziger Verkehrsbetriebe (LVB) GmbH;Ja;https://opendata.leipzig.de/dataset/8803f612-2ce1-4643-82d1-213434889200/resource/b38955c4-431c-4e8b-a4ef-9964a3a2c95d/download/gtfsmdvlvb.zip;https://opendata.leipzig.de/dataset/lvb-fahrplandaten;[email protected];Nein
MVV;cc-by;Münchner Verkehrs- und Tarifverbund GmbH;Nein;http://cms.opendata-oepnv.de/dataset/c231ec57-efb5-44fd-8bd6-7deb86cfcc60/resource/f443bf5d-ae07-42e3-85b8-da757159620b/download/gtfs-mvv-01-2020.zip;https://www.opendata-oepnv.de/dataset/soll-fahrplandaten-mvv;;Nein
naldo;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/naldo.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
NWL;CC BY 4.0;NWL;Nein;https://cms.opendata-oepnv.de/dataset/89892705-b6e7-4ffc-977c-2ba9b86dde46/resource/44f7d39c-02d7-4c2e-8efe-1046e5e959a2/download/gtfs-nwl-20200302.zip;https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nwl/startseite?tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-nwl&tx_vrrkit_view%5Bdataset_formats%5D%5B0%5D=ZIP&tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View;;Ja
OstalbMobil;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/oam.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
RAB;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/rab.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
RBS;dl-de/by-2.0;Datensatz der NVBW GmbH;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_ohne_Liniennetz/rbs.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-ohne-liniennetz/;[email protected];Ja
Rexer;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/rxr.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
RNV;dl-de/by-2.0;Rhein-Neckar-Verkehr GmbH (rnv);Ja;https://opendata.rnv-online.de/sites/default/files/rnv-GTFS_134.zip;https://opendata.rnv-online.de/dataset/gtfs-general-transit-feed-specification;[email protected];Ja
RVS;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/rvs.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
SBAHN-S;cc-by-4.0;Deutsche Bahn AG;Ja;http://download-data.deutschebahn.com/static/datasets/sbahn_stuttgart_gtfs/017_001_S-Bahn_Stuttgart_J19.zip;https://data.deutschebahn.com/dataset/data-s-bahn-stuttgart-gtfs;[email protected];Ja
SBG;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/sbg.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
SWEG;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/sweg.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
SPNV-BW;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/bwspnv.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/
SWEG;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/sweg.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/
SWHN;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/swhn.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
SWU;ODbL-1.0;;Ja;https://gtfs.swu.de/daten/SWU.zip;https://www.swu.de/privatkunden/service/mobilitaet/gtfs-daten/;[email protected];Ja
TGO;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/tgo.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
TUTicket;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/tuticket.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
VAGFR;dl-de/by-2.0;VAG Freiburg;Ja;https://www.vag-freiburg.de/fileadmin/gtfs/VAGFR.zip; https://www.vag-freiburg.de/fahrplan/fahrplanauskunft;[email protected];Nein
VBB;cc-by;Verkersverbund Berlin-Brandenburg GmbH;Nein;https://www.vbb.de/media/download/2029;https://www.vbb.de/unsere-themen/vbbdigital/api-entwicklerinfos/datensaetze;;Nein
VGC;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/vgc.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
VGF;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/vgf.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
VGN;cc-by-3.0-de;VGN – Ver­kehrs­ver­bund Groß­raum Nürn­berg GmbH;Ja;http://www.vgn.de/opendata/GTFS.zip;https://www.vgn.de/web-entwickler/open-data/;[email protected];Nein
VMT;cc-by-nd;VMT GmbH;Ja;https://www.vmt-thueringen.de/fileadmin/user_upload/Open_Data/VMT_GTFS.zip;https://www.vmt-thueringen.de/auskunft/open-data/;;Nein
VRN;cc-by;Verkehrsverbund Rhein-Neckar GmbH;Nein;https://www.vrn.de/mam/service/downloads/vrn_gtfs.zip;https://www.vrn.de/service/entwickler/gtfs/index.html;[email protected];Ja
VRR;cc-by;Verkehrsverbund Rhein-Ruhr AöR;Nein;https://openvrr.de/dataset/c415abd6-3b63-4a1f-8a17-9b77cf5f09ec/resource/a1e4fad2-fc11-4cd6-bb47-4bec24424f7c/download/2019_06_03_google_transit_verbundweit_mit_evu.zip;https://www.openvrr.de/dataset/gtfs;;Nein
VRS;cc-by;Verkehrsverbund Rhein-Sieg GmbH;Ja;http://download.vrsinfo.de/gtfs/google_transit.zip;https://www.vrsinfo.de/fahrplan/oepnv-daten-fuer-webentwickler.html;[email protected];Nein
VPE;dl-de/by-2.0 / Shapes ODbL;Datensatz der NVBW GmbH, Shapes (C) OpenStreetMap Mitwirkende;Ja;https://www.nvbw.de/fileadmin/nvbw/open-data/Fahrplandaten_mit_Liniennetz/vpe.zip;https://www.nvbw.de/aufgaben/digitale-mobilitaet/fahrplandaten-mit-liniennetz/;[email protected];Nein
VVS;cc-by;Verkehrs- und Tarifverbund Stuttgart Gmbh;Ja;https://www.openvvs.de/dataset/e66f03e4-79f2-41d0-90f1-166ca609e491/resource/bfbb59c7-767c-4bca-bbb2-d8d32a3e0378/download/vvs_gtfs.zip;https://www.openvvs.de/dataset/gtfs-daten;;Ja
33 changes: 33 additions & 0 deletions config/gtfs-rules/VGC.rule
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## Fix missing lat/lon:

# grep "000000" -r */stops.txt
# vgc.gtfs/stops.txt:de:08235:10449:0:4,"Nagold, Uferstr.",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:10451:0:3,"Nagold, Kreuzertal",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:3181:0:3,"Speßhardt, Friedhof",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:5660:2:3,"Neuenbürg, Eyachbrücke",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:5660:2:4,"Neuenbürg, Eyachbrücke",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:7330:0:3,"Wildberg, Bildungszentr.",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:7330:0:4,"Wildberg, Bildungszentr.",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:7330:0:5,"Wildberg, Bildungszentr.",0.000000,0.000000
# vgc.gtfs/stops.txt:de:08235:7330:0:7,"Wildberg, Bildungszentr.",0.000000,0.000000
# vgc.gtfs/stops.txt:gen:8235:10263:0:3,"Unterschwandorf, Löwen",0.000000,0.000000
# vgc.gtfs/stops.txt:gen:8235:2730:0:3,"Enzkl., Adventure Golfpark",0.000000,0.000000
# vgc.gtfs/stops.txt:gen:8235:2730:0:4,"Enzkl., Adventure Golfpark",0.000000,0.000000

# Guessing de:08235:10449:0:4,"Nagold, Uferstr.
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:10449:0:4"}, "update":{"stop_lat":"48.54923", "stop_lon":"8.71451"}}
# de:08235:10451:0:3,"Nagold, Kreuzertal", Location via [email protected]
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:10451:0:3"}, "update":{"stop_lat":"48.551247", "stop_lon":"8.737486"}}
# Speßhardt location via [email protected]
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:5660:2:3"}, "update":{"stop_lat":"48.714458", "stop_lon":"8.690857"}}
# Eyachbrücke location via [email protected]
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:5660:2:3"}, "update":{"stop_lat":"48.816358", "stop_lon":"8.577080", "stop_id":"de:08235:5660:2:1"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:5660:2:4"}, "update":{"stop_lat":"48.816326", "stop_lon":"8.577326", "stop_id":"de:08235:5660:2:2"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:7330:0:3"}, "update":{"stop_lat":"48.6241902615607", "stop_lon":"8.73791091051151"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:7330:0:4"}, "update":{"stop_lat":"48.6241812690453", "stop_lon":"8.73791095696822"}}
# Coordinates are estimates
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:7330:0:5"}, "update":{"stop_lat":"48.6241812690453", "stop_lon":"8.73791095696822"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08235:7330:0:7"}, "update":{"stop_lat":"48.6241812690453", "stop_lon":"8.73791095696822"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"gen:8235:10263:0:3"}, "update":{"stop_lat":"48.54465837060639", "stop_lon":"8.671057378464063", "stop_id":"de:08235:10263:0:3"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"gen:8235:2730:0:3"}, "update":{"stop_lat":"48.66741682564984", "stop_lon":"8.474423251273537", "stop_id":"de:08235:2730:0:3"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"gen:8235:2730:0:4"}, "update":{"stop_lat":"48.66741682564984", "stop_lon":"8.474423251273537", "stop_id":"de:08235:2730:0:4"}}
Loading

0 comments on commit e4f7b08

Please sign in to comment.