Support for TCP Searches #192
Conversation
…ible (e.g., if the connection has to go through firewalls); this can be done in those cases where server address is simply known, using combination of EPICS_PVA_ADDR_LIST and EPICS_PVA_SERVER_PORT variables
|
✅ Build pvAccessCPP 1.0.80 completed (commit 2da7698c75 by @sveseli) |
…T environment variable on both server and client side
sveseli
changed the title
There was a problem hiding this comment.
I can't comment on the PVA C++ specifics, I'm simply not familiar with it.
But as for configuring direct TCP connections to avoid the UDP searches:
PVXS and the new Java implementation use EPICS_PVA_NAME_SERVERS to set a list of TCP addresses. The "name server" in here is a bit misleading. You could of course list an actual name server IP, but to the client it just means that it sends a search request via TCP. The message handling code on the server is really the same, no matter if a message is received via UDP or TCP.
In CA, I think it was similar: EPICS_CA_NAME_SERVERS lists TCP addresses for searches.
So with EPICS_PVA_NAME_SERVERS="IP1 IP2 IP3" the client will send searches to those 3 IP addresses. A name server might reply with a search response that lists the actual IOC, but if the address is already an IOC, that IOC can return a search response with zero for the IP address to indicate: Continue on this TCP connection for the data exchange.
In more detail, EPICS_PVA_NAME_SERVERS may contain "IP:port" to list a TCP address with port. See https://github.com/epics-base/pvAccessCPP/wiki/Protocol-Messages#cmd_search-0x03 for details.
|
So with PVXS and the new java lib, this is how you would configure TCP-only searches: If only setting EPICS_PVA_NAME_SERVERS, it will contact those TCP addresses plus still send UDP searches. |
|
I feel bound to point out that PVXS already has a more complete "direct connection" (aka. bypassing the search phase entirely). One usage is in Also, fair warning. As I recall pvAccessCPP does not correctly handle reconnection of "direct" channels. So imo. it is not suitable for use with subscriptions specifically and may cause problems with long running clients generally. |
|
From what I have seen, before this PR direct connection to channels was not possible at all, as the code relied entirely on UDP searches, and if those are not resulting in channel discovery, there would be no connection made. In my tests, this PR handles monitor re-connections correctly, as long as the server comes up on the same machine and same port, regardless of whether UDP search is available or not. |
The Java PVA lib contains a command line demo, https://github.com/ControlSystemStudio/phoebus/blob/master/core/pva/src/test/java/org/epics/pva/server/SearchMonitorDemo.java It takes list of PVs and their IP:port on the command line and then replies to searches for those PVs with the provided name. That's of course only usable as a demo for one or two PVs, but it allows testing The plan was somewhat like this: Extend the channel finder's IOC tool (reccaster?) to provide the TCP port. I think that's been completed. |
|
Thanks Kay. Given that the PVA protocol has the ability for a name-server to query the servers for their complete list of PV names I don't think an oracle for PV names would be essential, although it might be good for performance reasons. There are 2 modes that I could see it using, maybe both at once: When the name-server sees a search request for a PV name that it doesn't recognize it would request that name through its client-side API, and would handle any response by asking the server that answered for all of its PVs. Alternatively/also it could monitor for beacons from new servers coming up, and proactively ask them for their names. The first mode would be needed for it to work with dynamic servers which could add new PVs at runtime, if it sees a response from a known server it would refresh the list of names for that server. I'm not quite sure how to handle dynamic servers that can drop PV names, somehow the name-server needs to know that's happened. I suspect the pvlist request can't be issued as a monitor, although that would be really helpful. It's important for the server to cache the PV names requested that it doesn't have a server for so it doesn't replicate every random request (the CA name-server has one). Supporting a configurable list of regexp's that match names to be forwarded and/or to not be forwarded might be a reasonable alternative to the Channel Finder to help with that. |
|
@mdavidsaver Unfortunately APS would need several FTE's of effort to convert all our DAQ software to the PVXS API, and we also rely on the plugin abilities currently unique to the pvAccess and pvDatabase libraries to be able to handle the high data rates coming from those DAQ systems. The data distributor plugin lets us accept updates from one data stream and fan them out to multiple clients to be processed in parallel. This PR provides the ability to connect through firewalls. I do agree with the need for tests. |
|
Andrew, you're correct, a name server might either rely on some type of name database (Channel Finder, Oracle, ...), or it might build that database itself. It could issue "pvlist" requests, or operate similar to the gateway by sending its own search request and then memorizing the reply. @shroffk those are options we could consider if we ever get back to working on a name server. |
|
We could consider a combination of those actions I have a very basic name server which uses ChannelFinder populated with recsync and the PVA port. I had imagined that the fall back mechanism for the name server if it fails to find the name resolution in CF would be to do its own name resolution search. |
|
Sorry about the accidental close |
|
✅ Build pvAccessCPP 1.0.99 completed (commit 1b3a3f5095 by @sveseli) |
|
Core Group: MAD will review this and provide feedback before the next meeting. |
|
✅ Build pvAccessCPP 1.0.103 completed (commit 9d7930478a by @sveseli) |
|
✅ Build pvAccessCPP 1.0.104 completed (commit c81093cb2c by @sveseli) |
|
✅ Build pvAccessCPP 1.0.105 completed (commit 84eb3d6400 by @sveseli) |
|
With @sveseli s recent pruning, I think I see what is going on wrt. name server handling. A couple of questions to check my understanding. Why are connections to name servers treated differently to other connections? Is each client context limited to connecting to one name server at a time? wrt. wrt. the design around eg. doing so stalls searching while name server(s) are offline. Compare with: |
… use timers; if multiple name servers are configured, connections to all of them are obtained at the same time
|
✅ Build pvAccessCPP 1.0.106 completed (commit 8fc112b9b2 by @sveseli) |
|
The blocking call for name server connections has been removed. Name server connections now use separate set of timers, so that they can be established at the same time. In the examples below I used non-existent hosts: $ time pvget foo Timeout foo real 0m5.021s user 0m0.011s sys 0m0.015s $ time EPICS_PVA_NAME_SERVERS="192.168.0.112:11111" pvget foo Timeout foo real 0m5.128s user 0m0.010s sys 0m0.023s $ time EPICS_PVA_NAME_SERVERS="192.168.0.112:11111 192.168.0.113:22222" pvget foo Timeout foo real 0m5.113s user 0m0.000s sys 0m0.021s Client can now establish connections to multiple name servers at the same time, and they are released as soon as they are no longer needed or aren't useful (hence methods that release those connections). Name server connections reuse existing classes (e.g. TransportRegistry, BlockingTCPConnector, etc.) as much as possible. |
|
✅ Build pvAccessCPP 1.0.107 completed (commit 5f53b171aa by @sveseli) |
|
@mdavidsaver Is there anything else you would like to see done or modified before this PR can be merged? |
|
@mdavidsaver @anjohnson Is this PR okay to be merged, or would you like to see some other changes? I would like to make sure that this goes into the next release, if that would be possible. |
No. There are several aspects of this proposed design which I do no like. eg. blocking |
The most recent commits (two months ago) addressed the issue of blocking connect you brought up. The connections are done via a separate set of timers. If there are other issues that need to be resolved, I would be happy to do this. |
Then what is going on with this // Wait to get transport if we do not have it
m_nsTransportEvent.wait(MAX_NS_TRANSPORT_WAIT_TIME);As for the rest. I have a limited amount of time which I can spend on EPICS community work, and many projects which ask for my attention. I would have an easier time reviewing your proposed changes if you would say more about what you are trying to achieve, and how you intend to achieve it. Correctly inferring intent from code is both difficult and error prone.
eg. Do you intend that an idle client will not maintain name server connections? Why not maintain NS connections continually? Are you are making a trade off of resource use vs. (re)connect latency? eg. Why are connections to name servers treated differently to other PVA server? Do you see this as a "need", a convenience, ...? |
If you look closely, the earlier code had an actual connect() in this thread. Right now a name server connect timer is scheduled that does the actual connect, and notifies corresponding event if connection was established. I used a short wait on this event just to see if connection can be established quickly, which improves search performance greatly when the name server responds, and does not hurt very much if name server does not respond, as we do not have channel connection anyways. I could certainly make sure that this wait happens only when the name server connection timer starts, rather than on every name server search attempt, but I suspect this will have a very little effect in terms of channel timeouts, etc. (see my examples of what happens when trying to connect to channels using non-existent machines).
This code is intended to work for a general case, where the name server connection is different from PVA Server connections. Rather than every client maintaining name server connections (and thus keeping those TCP sockets alive), I chose to close those connections as soon as I did not need them in order to preserve resources. There is obviously a trade off here in terms of resources and re-connection latency. One can perhaps argue one way or another as to what is more important, but given that name server can have hundreds or even thousands of client connections at any given time, I decided to prioritize resources, rather than to worry about reconnecting a bit faster every once in a while.
This stems partly from the fact that in general connections to name servers are different from connections to PVA servers in the sense we use them only in the discovery phase and also (for the reasons mentioned above) I chose to close them as soon as possible. Another reason is trying to minimize amount of the new code and impact of the required changes to the existing code, and making sure that no matter what happens with the name server connections, those do not affect client behavior and existing PVA server connections. I do appreciate the feedback/suggestions and I am willing to make any changes that are feasible and do not require major rewrite of the existing code. |
|
✅ Build pvAccessCPP 1.0.110 completed (commit a6f58f8049 by @sveseli) |
|
Meeting: AIs on SV:
|
This PR has been modified from the original (support for direct tcp connections) to include support for tcp searches.
Server side changes:
Client side changes:
Here are some examples of how things work. In terminal 1, we run test server using non-default ports:
In terminal 2, on a second machine, we run client. If we do not specify anything, client fails to connect:
If we specify broadcast port, channel connects using udp discovery:
If we specify EPICS_PVA_NAME_SERVERS variable, tcp search will result in channel connection: