Unable to load gtfs with missing required fields to fix it using the transformer-cli

[robbi5]

Summary:

I have a broken GTFS file with missing coordinates (stop_lat, stop_lon) on some stops in the stops.txt. To fix it, I wanted to use the transformer-cli with the update operation.
I've hoped that I could simply add them by using the transformer-cli with rules like this one:

{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08237:4004"}, "update":{"stop_lat":"48.474133", "stop_lon":"8.488702"}}

Sadly this failes, because transformer-cli aborts with an MissingRequiredFieldException: missing required field: stop_lat

Steps to reproduce:

GTFS from http://ubahndepot.com/storage/opendata/gtfs-scrapes/vgc-calw/gtfs-vgc-calw-2019-03-07--2019-06-08.zip as vgc.zip

rules file vgc.rule:

## Fix missing lat/lon:
# de:08237:4004,Dornstetten Hochgericht,,,,nses/7210214_26.json
# de:08237:4506,Pfalzgrafenweiler Sportanlagen,,,,nses/7210214_26.json
# de:08237:4645,Bösingen Ortsausgang,,,,nses/7210214_26.json
# de:08237:5122,Eutingen im Gäu Daimlerstraße,,,,nses/7210327_74.json
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08237:4004"}, "update":{"stop_lat":"48.474133", "stop_lon":"8.488702"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08237:4506"}, "update":{"stop_lat":"48.530459", "stop_lon":"8.577677"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08237:4645"}, "update":{"stop_lat":"48.538909", "stop_lon":"8.596061"}}
{"op":"update", "match":{"file":"stops.txt", "stop_id":"de:08237:5122"}, "update":{"stop_lat":"48.47754", "stop_lon":"8.74097"}}

Command: java -Xmx6g -jar onebusaway-gtfs-transformer-cli.jar --transform=vgc.rule vgc.zip vgc.fixed.zip

Expected behavior:

transformer-cli maybe warns about the missing required fields, but then allows the update operation to happen and returns a GTFS file with filled stop_lat, stop_lon fields.

Observed behavior:
transformer-cli doesn't even load the gtfs because stop_lat is a required field:

2019-03-24 10:51:45,959 INFO  [GtfsTransformerMain.java:191] : input path: vgc.zip
2019-03-24 10:51:46,084 INFO  [GtfsTransformerMain.java:196] : output path: vgc.fixed.zip
2019-03-24 10:51:46,091 INFO  [GtfsTransformer.java:192] : reading gtfs from vgc.zip
2019-03-24 10:51:46,092 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Agency
2019-03-24 10:51:46,099 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Block
2019-03-24 10:51:46,099 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.ShapePoint
2019-03-24 10:51:46,099 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Note
2019-03-24 10:51:46,099 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Area
2019-03-24 10:51:46,099 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Route
2019-03-24 10:51:46,125 INFO  [GtfsReader.java:178] : reading entities: org.onebusaway.gtfs.model.Stop
org.onebusaway.csv_entities.exceptions.CsvEntityIOException: io error: entityType=org.onebusaway.gtfs.model.Stop path=stops.txt lineNumber=619
	at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:161)
	at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:120)
	at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:115)
	at org.onebusaway.gtfs.serialization.GtfsReader.run(GtfsReader.java:180)
	at org.onebusaway.gtfs.serialization.GtfsReader.run(GtfsReader.java:168)
	at org.onebusaway.gtfs_transformer.GtfsTransformer.readGtfs(GtfsTransformer.java:194)
	at org.onebusaway.gtfs_transformer.GtfsTransformer.run(GtfsTransformer.java:157)
	at org.onebusaway.gtfs_transformer.GtfsTransformerMain.runApplication(GtfsTransformerMain.java:247)
	at org.onebusaway.gtfs_transformer.GtfsTransformerMain.run(GtfsTransformerMain.java:106)
	at org.onebusaway.gtfs_transformer.GtfsTransformerMain.main(GtfsTransformerMain.java:85)
Caused by: org.onebusaway.csv_entities.exceptions.MissingRequiredFieldException: missing required field: stop_lat
	at org.onebusaway.csv_entities.schema.AbstractFieldMapping.isMissingAndOptional(AbstractFieldMapping.java:100)
	at org.onebusaway.csv_entities.schema.DefaultFieldMapping.translateFromCSVToObject(DefaultFieldMapping.java:43)
	at org.onebusaway.csv_entities.IndividualCsvEntityReader.readEntity(IndividualCsvEntityReader.java:131)
	at org.onebusaway.csv_entities.IndividualCsvEntityReader.handleLine(IndividualCsvEntityReader.java:98)
	at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:157)
	... 9 more

Platform:

For reproducibility reasons I've tested it with onebusaway-gtfs-transformer-cli v1.3.63 and java 1.8.0_202 on macOS 10.14.

java version "1.8.0_202"
Java(TM) SE Runtime Environment (build 1.8.0_202-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b08, mixed mode)

Normally, I'm using the HSL digitransit stack, namely the OpenTripPlanner-data-container. This uses the docker base container openjdk:8-jre and onebusaway-gtfs-transformer-cli v1.3.9:

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)