Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-build linkages and egress tables for additional street modes via TransportNetworkConfig #862

Merged
merged 3 commits into from
Feb 7, 2023

Conversation

abyrd
Copy link
Member

@abyrd abyrd commented Feb 3, 2023

This is a user-configurable version of 010970c in #786.

The user can add "buildGridsForModes": ["BICYCLE", "CAR"] to the TransportNetworkConfig to pre-build linkages and distance tables and serialize them in the network. This should alleviate the problem of hundreds of regional workers all building the same distance tables (though not during some scenario applications, which will require additional optimizations).

Currently there is no way to inject TransportNetworkConfig during a bundle upload. We must manually edit the network config JSON file on the backend server (in ~/cache/bundles/networkid.json) and aws s3 cp to overwrite the copy on S3 before the network is built by a single point worker.

@abyrd
Copy link
Member Author

abyrd commented Feb 3, 2023

This appears to be working. After creating a bundle on production I edited its scenario JSON cache/bundles/63dcf6f7fdc3560d3366d09f.json to contain "buildGridsForModes":["CAR"] then copied it to s3 with aws s3 cp cache/bundles/63dcf6f7fdc3560d3366d09f.json s3://scenario-editor-prod-bundles/63dcf6f7fdc3560d3366d09f.json.

I then started two single point analyses in different tabs with worker versions v6.8 and v6.8-5-g8a509ae. Worker logs show the v6.8 worker computing only WALK data and returning a result quickly. Worker logs on v6.8-5-g8a509ae show it computing both WALK and CAR linkages and egress tables (very slowly for CAR).

So existing workers are not bothered by the new field, and new workers build data for the requested modes.

New workers log the mode when they make linkages or egress tables:

INFO  c.c.r.s.LinkedPointSet - Linking pointset to street network for mode CAR...
INFO  c.c.r.s.LinkedPointSet - Done. Linked 297k of 297k PointSet points to streets for mode CAR.
INFO  c.c.r.s.EgressCostTable - Computed new stop-to-point tables from 100 of 7k transit stops for mode CAR.

@abyrd
Copy link
Member Author

abyrd commented Feb 6, 2023

To explain the larger blocks of code removed/moved: The first 10 lines or so of buildNetworkFromConfig have been factored out into a new method loadNetworkConfig and called from buildNetwork, such that the network config is availble in the outer buildNetwork scope and not just inside buildNetworkFromConfig. It's needed in the outer scope to read the additional modes in proprty buildGridsForModes. network.scenarioId = networkId was already being set in that outer scope, so there was no need to pass the network ID into the buildNetworkFromConfig method. Most of the other changes are just comments that should reduce the amount of digging needed to understand behavior in the event of future optimizations.

Copy link
Member

@ansoncfit ansoncfit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes and updated comments.

@abyrd abyrd enabled auto-merge February 7, 2023 02:49
@abyrd abyrd merged commit 995635a into dev Feb 7, 2023
@abyrd abyrd deleted the prebuild-non-walk branch February 7, 2023 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants