sf + webhost integration problems: port binding (kestrel), primary -> secondary transition

[cperras]

Hi -

I'm following the pattern shown here:
• Create web-host (often with Kestrel).
• Register service-fabric services (asynchronously)
• Run the web-host

However, the lifetime of the web-host persists beyond the sf-services, and the web-host stays open until the process terminates. Since the web-host is already initialized when CreateServiceInstanceListeners() is called, ICommunicationListener.OpenAsync (and CloseAsync) are essentially no-ops.

This is probably not the intended sf pattern; instead the listener should be started inside ICommunicationListener.OpenAsync(), and stopped in CloseAsync(). An issue I'm seeing now is that a websocket based stateful service instance went from primary -> secondary, and since all the websocket clients remained connected to the secondary, requests failed since they weren't routed to the primary (if the connections closed, the service-instance would be re-resolved, and everything would work as expected).

However, it’s not immediately clear how, or whether, I should make this change as it doesn’t seem to fit the aspnet-core web-host pattern. In particular, if I stopped (disposed) the web-host in CloseAsync(), it's problematic to restart it again since all the process initialization logic in Startup.cs will be triggered (with DI services re-registering, etc etc).

As a hack, I can detect the primary -> secondary transition and manually close all the websocket connections. But, it remains a concern that all secondaries (not just the ones which transitioned to primary) will have the port open and actively listening sf (I don't want read-only secondaries).

To compound the problems, since I use Kestrel, each secondary will exclusively bind to the port, preventing other instances (eg: primaries) from starting.

How do I best use aspnet core within service fabric to prevent, or at least mitigate, the above problems?

It seems that moving to WebListener is a good first step. At that point, the processes can at least share ports, and different service-instances can bind to specific urls (eg: with partitionIds and/or guids ) so > 1 primary/secondary can run on a node (presumably http.sys or some subsystem routes requests to the correct service-instance even if they have the same port) ala your service-registry.

Has anyone else ran into these issues, or tried using Kestrel in service fabric (in a production, or near-production, capacity)?

thanks!
--craig

[dazhao-msft]

Hi Craig,

Sorry for slow response. Currently I'm not actively working on Service Fabric.

Service Fabric team has released a NuGet package using WebListener, which creates web host per replica/instance. You can try it out to see if that better fits your needs.

Let me know if you have further questions. I can help redirect the questions to the right contact.

Thanks,
-David