How reliable is sync=true API call, especially against the mainnet ?

[SiriusDely]

After testing using sync=true against testnet for a while, we decided to deploy an instance of flow-wallet-apito connect to the mainnet.

For example for the endpoint to create an account: POST /v1/accounts?sync=true

When the wallet-api x testnet can, most of the time, successfully response under 30/20 seconds, however the wallet-api x mainnet always returns 503 - Service Unavailable in under 60 seconds or less, _doesn't matter if the time-out configuration of the wallet-api deployment is increased to a value something like 900 seconds.

However the account creation is indeed successfully created in the background by checking it from GET /v1/accounts.

So the question is: How reliable is sync=true API call, especially against the mainnet ? Do we even have a chance to use synchronized call on the mainnet (production)?

Thank you.

This is all the log that we have for now:

Error
2021-11-04T02:10:41.733689Z POST 503 975 B 18.5 s curl/7.64.1  https://XXX/v1/accounts?sync=true
The request failed because either the HTTP response was malformed or connection to the instance had an error.

Default
2021-11-04T02:10:42.862632Z [SERVER] 2021/11/04 02:10:42 main.go:74: Starting server (v0.6.2)...

Default
2021-11-04T02:10:43.420761Z [SERVER] 2021/11/04 02:10:43 main.go:231: Server listening on :3000

Default
2021-11-04T02:10:43.439882Z [EVENT-POLLER] 2021/11/04 02:10:43 listener.go:145: started

This is deployed on the GCP Cloud Run.

[latenssi]

Hi @SiriusDely and thank you for the writeup! The sync feature is really not meant for production use and therefore we have no information on the reliability of it.

That said the timeout you are experiencing seems to happen on the HTTP connection between the wallet and the HTTP client calling it. I base this guess on the fact that the wallet service itself never responds with a 503, so it must be the calling client setting that response status. For example if you are doing the POST /v1/accounts?sync=true with curl it may have max-time parameter set, more info here: https://unix.stackexchange.com/questions/94604/does-curl-have-a-timeout/94612. Each HTTP client software probably has a different way of setting that.

You were asking for webhooks for the job statuses before. Would webhooks allow you to not use sync?

[SiriusDely]

Thank you for quick response @latenssi .

Have tried to increase the curl timeout, however still with no luck. curl --request POST --connect-timeout 900 --max-time 900 'https://XXX/v1/accounts?sync=true'

You were asking for webhooks for the job statuses before. Would webhooks allow you to not use sync?

Yes, this is exactly the reason why I was asking for the webhooks. However since it is not yet available, it makes me wondering for the available alternatives or recommended approaches here.

I can list these two below, for which I am asking your opinion on the better approach, or probably you have other approaches that is even better, please let me know:

1. Poll the related endpoints for job and transaction status, ie. GET /v1/jobs/:id and GET /v1/transactions/:id
2. Only check those endpoints above (1) when corresponding logic is needed to be executed and related/dependent jobs/transactions/account creations is in-progress. Which also requires some kind of a queue/retry system to defer this logic whenever the dependent jobs is not finished yet.
3. Combination of (1) & (2) above.
4. ... ?

Thank you.

The sync feature is really not meant for production use

It looks like we need to put this on the documentation somewhere (or am I missing it?). I will try to do this. Where do you think a suitable place to put it, on the readme or the openapi.yaml or somewhere else?

[latenssi]

It looks like we need to put this on the documentation somewhere (or am I missing it?). I will try to do this. Where do you think a suitable place to put it, on the readme or the openapi.yaml or somewhere else?

I think the openapi.yml would make most sense as that is where the sync option is presented. Thank you for offering to add this, it would be much appreciated if you could make a PR adding this @SiriusDely! 👍

I can list these two below, for which I am asking your opinion on the better approach, or probably you have other approaches that is even better, please let me know

Option 3 is what we assumed users of this software would have to implement: poll the endpoints when needed.

We are currently discussing the webhooks feature and its priority. It would help us if you could describe the way you would like webhooks to work 🙏 And anyone else reading this can of course chime in as well! Please add all webhooks related conversation to #180

EDIT: added call to move webhook discussion to #180

[latenssi]

We will investigate the issue with sync calls.

[SiriusDely]

I think the openapi.yml would make most sense

Sure, will send a PR soon.

Option 3 is what we assumed users of this software would have to implement: poll the endpoints when needed.

OK, this is much clearer now.

if you could describe the way you would like webhooks to work

I have put some comments on 180.

We will investigate the issue with sync calls.

Thank you @latenssi

[latenssi]

@SiriusDely Are you connecting directly to the wallet API with curl or do you have a reverse proxy (or something similar) in between?

[SiriusDely]

@latenssi direct connection to the cloud run deployment, without proxy in between, I have tried both using the custom domain and the url provided by cloud run, ie. https://wallet-api-xxx-uc.a.run.app, just in case, however still getting the same Service Unavailable.

[SiriusDely]

one more thing that I've just noticed, it looks like the 503 - Service Unavailable response returns just after (or before?) the transaction is actually completed. I tried by invoking POST /v1/accounts?sync=true from one terminal and then watch GET /v1/accounts?limit=1 on another terminal. Screen recording provided below.

Screen.Recording.2021-11-10.at.13.11.03.mov

[latenssi]

Thank you @SiriusDely for this comprehensive display! 👍

[nanuuki]

@SiriusDely A configurable timeout was added in #211, please try the updated version (and adjust FLOW_WALLET_SERVER_REQUEST_TIMEOUT to fit your needs) in case you are still using the sync calls :)

[SiriusDely]

@nanuuki we are in progress of integration the webhook, however will surely test this update, will let you know when this fixes the problem reported on this issue. Thank you! Really appreciate this.

[SiriusDely]

can confirm this issue is resolved, thank you @nanuuki @latenssi !