Backoff Reconnect #6358

lee-at-zoo-corp · 2025-04-16T20:42:24Z

Backoff reconnect behavior added (Fibonacci sequence used due to nice time scaling properties)
Translate WebSocket error numbers to text on the loading screen
Detect disconnections and reconnect
Restore camera state on reconnect even on sudden disconnects
Add a RestartRequest event that the application can fire at any point to restart the engineConnection
Add E2E stream pause test
Lifted more logic from EngineStream into engineStreamMachine
Added a slew of debug messages to network code startup

vercel · 2025-04-16T20:42:29Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
modeling-app	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 15, 2025 3:52pm

qa-wolf · 2025-04-16T20:42:31Z

QA Wolf here! As you write new code it's important that your test coverage is keeping up.
Click here to request test coverage for this PR!

codecov · 2025-04-16T21:04:45Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.41%. Comparing base (3a23ec1) to head (3dc4e0f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6358   +/-   ##
=======================================
  Coverage   85.41%   85.41%           
=======================================
  Files         109      109           
  Lines       47482    47482           
=======================================
  Hits        40559    40559           
  Misses       6923     6923

Flag	Coverage Δ
rust	`85.41% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

src/machines/engineStreamMachine.ts

…s for us

…on and restoral

src/components/EngineStream.tsx

src/machines/engineStreamMachine.ts

pierremtb

Alright I parsed through it, I can't pretend that I understand it all but it all seems to make sense. I have one request to look at before merging: the use of the TEST env var, see comment at the top of EngineStream.tsx about using an existing helper.

Going to test locally now!

e2e/playwright/snapshot-tests.spec.ts-snapshots/theme-persists-1.png

src/components/EngineStream.tsx

src/lang/std/engineConnection.ts

graphite-app · 2025-05-15T15:28:46Z

src/lang/std/engineConnection.ts

+      // We should eventually only have 1 restoral call.
+      if (this.idleMode) {
+        await this.sceneInfra?.camControls.restoreRemoteCameraStateAndTriggerSync()
+      } else {
+        // NOTE: This code is old. It uses the old hack to restore camera.
+        console.log('call default_camera_get_settings')
+        // eslint-disable-next-line @typescript-eslint/no-floating-promises
+        await this.sendSceneCommand({
+          // CameraControls subscribes to default_camera_get_settings response events
+          // firing this at connection ensure the camera's are synced initially
+          type: 'modeling_cmd_req',
+          cmd_id: uuidv4(),
+          cmd: {
+            type: 'default_camera_get_settings',
+          },
+        })
+      }


The await keyword is missing in the if (this.idleMode) branch when calling this.sceneInfra?.camControls.restoreRemoteCameraStateAndTriggerSync(). This could lead to timing issues since the function appears to be asynchronous. Both branches should consistently use await to ensure proper sequencing of operations during camera restoration.

// Should be: if (this.idleMode) { await this.sceneInfra?.camControls.restoreRemoteCameraStateAndTriggerSync() } else { // ... }

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

Wow the first time it's wrong? Both branches clearly use await?

pierremtb

Tested locally on macOS and Windows. LGTM!

And thanks for the changes!

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from 385d799 to dd8b870 Compare April 16, 2025 20:46

vercel bot deployed to Preview April 16, 2025 21:06 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from feb295a to dbdbc8d Compare April 18, 2025 03:36

vercel bot deployed to Preview April 18, 2025 04:01 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from dbdbc8d to f01a092 Compare April 21, 2025 17:20

vercel bot deployed to Preview April 21, 2025 17:33 View deployment

paultag closed this Apr 22, 2025

paultag reopened this Apr 22, 2025

paultag marked this pull request as ready for review April 22, 2025 18:32

paultag marked this pull request as draft April 22, 2025 18:34

graphite-app bot reviewed Apr 22, 2025

View reviewed changes

src/machines/engineStreamMachine.ts Show resolved Hide resolved

vercel bot deployed to Preview April 22, 2025 18:44 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from bfec3c4 to 8fcf4f1 Compare April 25, 2025 14:16

vercel bot deployed to Preview April 25, 2025 14:28 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from 8fcf4f1 to aa92c11 Compare April 28, 2025 20:47

vercel bot deployed to Preview April 28, 2025 20:58 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from aa92c11 to 3dc4e0f Compare April 28, 2025 21:39

vercel bot deployed to Preview April 28, 2025 21:51 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from 3dc4e0f to 3afd31b Compare April 30, 2025 14:20

vercel bot deployed to Preview April 30, 2025 14:33 View deployment

lee-at-zoo-corp force-pushed the lee-at-zoo-corp/backoff-reconnect branch from 3afd31b to f54582f Compare April 30, 2025 19:38

lee-at-zoo-corp added 6 commits April 30, 2025 15:38

Add a Stop command that does a bit of cleanup in the engineStateMachine

0530d12

Major network code startup refactor; backoff reconnect; many more log…

77847b7

…s for us

Save camera state after every user interaction in case of disconnecti…

3b1ece3

…on and restoral

Translate basic WebSocket error numbers into useful text

fac0804

Add a RestartRequest event to the engineCommandManager

6c07ae8

Adjust for a rebase

cc0aef6

Merge branch 'main' into lee-at-zoo-corp/backoff-reconnect

d4c7e85

graphite-app bot reviewed May 2, 2025

View reviewed changes

src/components/EngineStream.tsx Outdated Show resolved Hide resolved

vercel bot deployed to Preview May 2, 2025 21:56 View deployment

pierremtb mentioned this pull request May 12, 2025

Connection failure did not reconnect #6402

Closed

pierremtb linked an issue May 12, 2025 that may be closed by this pull request

Improved error when you cant get a stream #6050

Closed

lee-at-zoo-corp added 2 commits May 14, 2025 13:39

Merge branch 'main' into lee-at-zoo-corp/backoff-reconnect

4017d1b

small issue

fb5c5cc

graphite-app bot reviewed May 14, 2025

View reviewed changes

src/machines/engineStreamMachine.ts Outdated Show resolved Hide resolved

vercel bot had a problem deploying to Preview May 14, 2025 17:53 Failure

lee-at-zoo-corp added 3 commits May 14, 2025 14:55

Fix tests

ffaa05f

Fix up idle test

eb9d177

One last fix

99863f1

vercel bot deployed to Preview May 14, 2025 19:58 View deployment

lee-at-zoo-corp added 2 commits May 14, 2025 16:54

fixes

4f64c32

Remove circ dep

ee590a0

vercel bot deployed to Preview May 14, 2025 21:48 View deployment

fix test

1f0c914

vercel bot deployed to Preview May 15, 2025 14:24 View deployment

pierremtb requested changes May 15, 2025

View reviewed changes

lee-at-zoo-corp added 2 commits May 15, 2025 11:23

use a const for time

b3296bc

TEST -> isPlaywright instead

da9d294

graphite-app bot reviewed May 15, 2025

View reviewed changes

vercel bot deployed to Preview May 15, 2025 15:39 View deployment

whoops

b8e6d53

pierremtb approved these changes May 15, 2025

View reviewed changes

vercel bot deployed to Preview May 15, 2025 15:52 View deployment

lee-at-zoo-corp merged commit 3f00e71 into main May 15, 2025
49 checks passed

lee-at-zoo-corp deleted the lee-at-zoo-corp/backoff-reconnect branch May 15, 2025 15:58

This was referenced May 15, 2025

Make idle setting 30sec in nightly #6972

Merged

[BUG]: Silent Stream (I think) Crash #6662

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backoff Reconnect #6358

Backoff Reconnect #6358

lee-at-zoo-corp commented Apr 16, 2025 •

edited

Loading

vercel bot commented Apr 16, 2025 •

edited

Loading

qa-wolf bot commented Apr 16, 2025

codecov bot commented Apr 16, 2025 •

edited

Loading

pierremtb left a comment

graphite-app bot May 15, 2025

lee-at-zoo-corp May 15, 2025

pierremtb left a comment •

edited

Loading

Backoff Reconnect #6358

Backoff Reconnect #6358

Conversation

lee-at-zoo-corp commented Apr 16, 2025 • edited Loading

vercel bot commented Apr 16, 2025 • edited Loading

qa-wolf bot commented Apr 16, 2025

codecov bot commented Apr 16, 2025 • edited Loading

Codecov Report

pierremtb left a comment

Choose a reason for hiding this comment

graphite-app bot May 15, 2025

Choose a reason for hiding this comment

lee-at-zoo-corp May 15, 2025

Choose a reason for hiding this comment

pierremtb left a comment • edited Loading

Choose a reason for hiding this comment

lee-at-zoo-corp commented Apr 16, 2025 •

edited

Loading

vercel bot commented Apr 16, 2025 •

edited

Loading

codecov bot commented Apr 16, 2025 •

edited

Loading

pierremtb left a comment •

edited

Loading