Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sjoin performance #831

Merged
merged 8 commits into from
Aug 14, 2023
Merged

Sjoin performance #831

merged 8 commits into from
Aug 14, 2023

Conversation

tiffanychu90
Copy link
Member

@tiffanychu90 tiffanychu90 commented Aug 14, 2023

segment speeds

  • performance improvements
  • bring over map_partitions from spatial accuracy to speeds pipeline and use this instead of dask.delayed where possible
  • find that some of the flipping of segments for inlining is still not correct. make this more accurate by moving when the np.flip occurs.
    • check the direction vector of 2 stops, and if it doesn't match segment direction, flip it. don't use the projected shape_meters as check for flipping.
  • rewrite how vp is triangulated - remove loops
  • rerun Mar-Jul 2023 entirety of pipeline. each month roughly takes 1-1.5 hrs
    • clean up logs to only include latest run of all months
  • adjust segment_speed_utils to use with map_partitions
    • instead of assembling the dfs within loops for delayed, we can assemble the df overall, and map the same function to the partitioned df.
  • remove scripts to patch in new columns like trip_instance_key...it's tested on full runs of the workflow now
  • Epic - Daskify RT segment speeds #592

@tiffanychu90 tiffanychu90 merged commit 066b8ff into main Aug 14, 2023
2 checks passed
@tiffanychu90 tiffanychu90 deleted the sjoin-performance branch August 14, 2023 16:41
@tiffanychu90 tiffanychu90 mentioned this pull request Aug 31, 2023
21 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant