Add vp dwell time / pilot Big Blue Bus speeds post alignment

[tiffanychu90]

vehicle positions in gtfs_funnel

• Add dwell time (seconds) and create a moving_timestamp_local column that captures when that vp starts moving again
• Difference between location_timestamp_local and moving_timestamp_local is dwell time
• Include this as part of gtfs_funnel, and this step should go before vp_condenser
  • TODO backfill all dates and make this available
  • TODO update later scripts to use this file / add this file into gtfs_analytics_data.yml so the swap is seamless
  • TODO vp_condenser should be backfilled too, because this will actually have a lot fewer vp to condense into a linestring
• Research Request - add dwell time for vehicle positions #1163

segment speeds

• pilot Big Blue Bus through the rest of the steps to align with Eric's speedmap methodology
• stand up new scripts that only focus on Big Blue Bus to see how pipeline scripts should change
• add 37_bbb_speeds_with_dwell notebook to quickly look at how Big Blue Bus results. tweak those scripts as necessary, and once it's good, put the content back into the main pipeline scripts.
• Research Request - Nearest neighbor comparison  #1162
• change how nearest neighbor outputs are saved. detailed methodology alignment breakdown:

1. new_nearest_10

• existing: nearest neighbor now grabs the one nearest neighbor vp and then grabs 2 points around that (using vp_idx) and adds all the information about geometry, location_timestamp_local, and then projects it onto the shape and gets stop_meters`
• alignment: grab the nearest 10 vp and save these vp_idx out only. grab 10 because there was no guarantee that the 1 point that was grabbed was before the stop. in the absence of knowing, let's grab a large enough list so we can definitely get 1 vp before the stop.
• TODO: backfill all dates once pilot checks out. this overwrites old files.

2. new_narrow_to_2

• existing: not an existing step, since we grabbed 1 vp only and populated the trio around it
• alignment: add in scheduled shape here, shapely.project each of them against the shape. we know where the stop falls on the shape (stop_meters), so now we can narrow down to the 2 vp that occur before and after the stop (keep the 2 vp that have the smallest positive and negative deltas from stop_meters).
• TODO: backfill all dates once pilot checks out. this creates a new file!

3. new_interpolate

• existing: interpolation is done between 2 location_timestamp_local columns converted into seconds.
• alignment: now that we have dwell time, we will take difference between a moving_timestamp_local (previous stop) and a location_timestamp_local (current stop). logic should be the time the bus starts moving from previous stop and before it idles at current stop. double check this step.
• TODO: backfill all dates once pilot checks out. this overwrites old files.

4. quick_bbb_speeds

• existing: no major changes here, simply combine steps in stop_arrivals_to_speeds and average_segment_speeds into one script for BBB to check results. Save out 2 files to use in exploratory notebooks to figure out how to better filter. Everything in 37_bbb_speeds_with_dwell is unfiltered so we can look at the full distribution of speeds
• alignment: this is our TODO
• TODO: align here, and everything can return into the existing stop_arrivals_to_speeds and average_segment_speeds scripts

[github-actions]

nbviewer URLs for impacted notebooks:

• rt_segment_speeds/37_bbb_speeds_with_dwell.ipynb

[github-actions]

nbviewer URLs for impacted notebooks:

• rt_segment_speeds/37_bbb_speeds_with_dwell.ipynb

[github-actions]

nbviewer URLs for impacted notebooks:

• rt_segment_speeds/37_bbb_speeds_with_dwell.ipynb

[github-actions]

nbviewer URLs for impacted notebooks:

• rt_segment_speeds/37_bbb_speeds_with_dwell.ipynb

[tiffanychu90]

@edasmalchi: can you look at the files I created for Big Blue Bus that are imported in 37_bbb_speeds_with_dwell.ipynb and use it in an exploratory notebook? I pulled out some examples, but I'll also look more!

[edasmalchi]

yeah!

[edasmalchi]

@tiffanychu90 looks like the trip-level speeds are substantially equivalent, but the aggregations vary a lot?

It's not the exact same time period so it's a little hard to compare, but even comparing either peak from the legacy speedmaps to the new combined peak, there's quite a difference. Good news though I think, if the trip speeds are so close it shouldn't be so hard to align the aggregations? Might be worth having separate am/pm in the new pipeline anyway, would allow us to keep presenting more information...

Will continue looking!

https://github.com/cal-itp/data-analyses/blob/eric-bbb-exploratory/rt_segment_speeds/38_bbb_speeds_legacy_compare.ipynb