RefEx section - Dataflow API error: The job graph is too large

[ivelin]

Congrats on the impressive SoTA results! 🥇

In following the steps to reproduce the tests, I stumbled into a couple of issues. Some were easy to resolve, but the one below is a blocker. Any suggestions would be appreciated. 🙏🏼

The data prep step in the RefEx section fails with the error below:

python -m pix2struct.preprocessing.convert_refexp   --data_dir=$PIX2STRUCT_DIR/data/refexp   --image_dir=$PIX2STRUCT_DIR/data/rico_images   --   --runner=DataflowRunner   --save_main_session   --project=$GCP_PROJECT   --region=$GCP_REGION   --temp_location=$PIX2STRUCT_DIR/data/temp   --staging_location=$PIX2STRUCT_DIR/data/staging   --setup_file=./setup.py

  File "/home/gitpod/miniconda3/envs/pix2struct/lib/python3.9/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
  File "/home/gitpod/miniconda3/envs/pix2struct/lib/python3.9/site-packages/apitools/base/py/base_api.py", line 603, in __ProcessHttpResponse
    raise exceptions.HttpError.FromResponse(
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/pix2struct-373300/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Sat, 31 Dec 2022 03:21:02 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '400', 'content-length': '228', '-content-encoding': 'gzip'}>, content <{
  "error": {
    "code": 400,
    "message": "(634623baaa978ce): The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.",
    "status": "INVALID_ARGUMENT"
  }
}
>