Skip to content
This repository has been archived by the owner on Jul 21, 2023. It is now read-only.

TimeoutException on SetAsync() #12

Open
rwallerms opened this issue Jun 12, 2020 · 9 comments
Open

TimeoutException on SetAsync() #12

rwallerms opened this issue Jun 12, 2020 · 9 comments

Comments

@rwallerms
Copy link

While running load tests against our service, we're noticing that call to SetAsync() will sometimes fail with a TimeoutException, with what appears to be a 5 minute timeout.

System.TimeoutException: This can happen if message is dropped when service is busy or its long running operation and taking more time than configured Operation Timeout.

Tried setting the CancellationToken to limit the duration of the call, but it looks like that is not used in this method.

service-fabric-distributed-cache/src/SoCreate.Extensions.Caching.ServiceFabric/ServiceFabricDistributedCache.cs

Line 80 in bf2db07

await proxy.SetCachedItemAsync(key, value, options.SlidingExpiration, absoluteExpireTime).ConfigureAwait(false);

Our load tests did not seem particularly intensive. They only pushed the machine to around 20% CPU.
@Ryanman
Copy link

Ryanman commented Jun 25, 2020

Looks like there's a new PR out that should help you narrow down the issue;

#13

We moved from Set to SetAsync() just a couple days ago have now experienced service failures ("Partition is in Quorum Loss") and the app's exit code appears to have been swallowed (""System.Hosting' reported Error for property 'CodePackageActivation:Code:EntryPoint:132315329573570487'.
There was an error during CodePackage activation.The process/container terminated with exit code:255.")

Did you find the source of the issue or are you still working?

@rwallerms
Copy link
Author

I was hoping someone from the SoCreate team would chime in with their opinion as to why this might happen and/or whether they can add support for using the CancellationToken to work around.

@Ryanman
Copy link

Ryanman commented Jun 26, 2020

I haven't gotten the patch yet either, I'll mention I recently was able to watch SetAsync() run through. When performing multiple set operations in a row, Async was hundreds/thousands of times slower than non-async set.

Perhaps there are speed benefits when you queue up these cache sets? I don't have any more information besides the fact I could sit there and watch how slow it was to set individual cache buckets asynchronously.

@lurock
Copy link
Contributor

lurock commented Jul 2, 2020

Have you run this against the latest version of the of the library? How many items are you adding to the cache and how large are the items? Do you have a load test example I could run to duplicate the issue?

@rwallerms
Copy link
Author

rwallerms commented Jul 2, 2020 via email

@lurock
Copy link
Contributor

lurock commented Jul 3, 2020

Please try version 2.1.2 that I have just pushed up. I found an threading issue causing some random exceptions. Let me know if this fixes your issues.

@rwallerms
Copy link
Author

rwallerms commented Jul 3, 2020 via email

@rwallerms
Copy link
Author

rwallerms commented Jul 6, 2020 via email

@mohammedjarwan
Copy link

Hi @lurock

we are using SetAsync() heavly it's too slow using vesion 2.1.3 , we are cahcing small data like {"numericId":"300000000000021","expirationTime":"2020-08-24T08:11:31.4126188Z"}

do u have any idea?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Ryanman @mohammedjarwan @lurock @rwallerms