-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Orleans streams fail to deliver messages on silo1 after silo2 shutdown #9126
Comments
I'm seeing the same issue with persistent (postgres) streams. As described by @FlorianMeinschad, it is rather sporadic and hard to pinpoint exactly under what circumstances consumers stop receiving messages. |
I was under the impression that Orleans was production-ready software. However, the bug (and the two comments above) indicate that it is not yet production-ready and that important functions (such as PubSub) do not work as indicated. This raises concerns about its suitability for enterprise solutions. |
To be fair, I think persistent streams for AdoNet is relatively new so I
can imagine there are still some bugs to be snuffed out. I’m going to wire
up the code to Azure Queues to validate whether it’s an issue with the
underlying AdoNet stream implementation.
…On Tue, Aug 27, 2024 at 1:08 AM Mathias Feitzinger ***@***.***> wrote:
I was under the impression that Orleans was production-ready software.
However, the bug (and the two comments above) indicate that it is not yet
production-ready and that important functions (such as PubSub) do not work
as indicated. This raises concerns about its suitability for enterprise
solutions.
—
Reply to this email directly, view it on GitHub
<#9126 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADL4KRXMXYLLQOZC7DPT7D3ZTQCUVAVCNFSM6AAAAABNDJEG6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGU4DENRQGU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I'm not sure if this is helpful, |
So in summary, there is currently no solution to get orleans streaming up and running without using Azure or AWS services? In our case we host the solution on our own servers. We have SQL servers and use the ADO.Net Orleans provider. |
@scalalang2 Thanks, that is actually helpful. Haven't seen much documentation about Orleans on AWS, so didn't think of trying it! In the meantime, I tried on Azure and it's working as expected with little to no changes to the actual domain code, so it does confirm that the issue is with the ADO.NET streaming provider. @feitzi I'm wondering whether it's possible to mix and match the tech stack, meaning everything else is ran on ADO.NET (actor state, reminders, etc) while PubSub is ran on Azure/AWS. Will give it a try later this week... |
We are encountering a critical issue with Orleans 8.2.0 streams that significantly impacts our application's functionality.
The problem was first observed in our production environment, and although there were no silo shutdowns or any relevant logs indicating failures, the stream functionality appears to stop working correctly after a period of time.
To investigate further, we created a small proof-of-concept (POC) application that can reproduce the issue most of the time.
The issue can be reproduced by shutting down a silo, which occasionally causes the subscriber grains to remain active (sometimes all subscriber grains are stopped), but they stop receiving stream messages.
We suspect a similar problem to the one mentioned in this issue #8540.
We are unsure if this is a result of incorrect usage of Orleans streaming features, but the behavior seems inconsistent and doesn't align with our expectations.
Could you kindly review our POC code and provide an analysis of the issue?
Any insights or guidance would be greatly appreciated.
Thank you in advance for your assistance.
Kind regards
POC Code
https://github.com/FlorianMeinschad/OrleansStreamPOC
POC steps (The issue may not occur every time; multiple attempts might be required):
http://localhost:5000/pub/5
http://localhost:5000/sub/3
http://localhost:5000/sub/3
http://localhost:5000/dashboard
Log of Silo2:
Log of Silo1 (no more publisher messages are received):
Sometimes the following exceptions are thrown:
Log of Silo2
Log of Silo1
The text was updated successfully, but these errors were encountered: