Allow for a more granular selection of poll period for certain signals

[jwlodek]

Currently, all the signals from the panda are polled every 0.1 seconds. Certain signals, however, can be more useful if they are polled faster. For example - if you want to use the PandA's logic to "trigger" a device that doesn't offer a hardware trigger/TTL option, you can do this right now with a CA link to one of the TTLOUT:OUT signals. However, in my experience this limits you to sub ~5 Hz trigger rates, since the PandA IOC polls everything at the same rate.

[AlexanderWells-diamond]

I'm not sure we have the ability to do what is being asked. The issue is that, currently, we poll the PandA for all changes since the last time we asked, using the GetChanges function. There is an enum to control what values the PandA reports back, but it isn't particularly fine-grained. See the enum defined here. If the field(s) you are interested in rapidly polling are reported by one of those options, then we should be able to do faster polling for specific groups. Otherwise I don't think we can offer anything better than simply increasing the global polling rate.

[jwlodek]

I think for my use case the most useful group to poll faster would be the BITS values: https://github.com/PandABlocks/PandABlocks-client/blob/c2c8c0b1d78f8317b18602e85984cab365b946d2/src/pandablocks/commands.py#L780, because I believe this includes all the block outputs that are presumably tied to your hardware via TTLOUT blocks.

[evvaaaa]

Maybe we could add new command line args to specify different poll periods for specific change groups. @coretl thoughts?

[jwlodek]

Just to see if I was hitting any kind of limitation in terms of the max scanning rate of the CA link mechanism from EPICS, I created a simple record that should produce a 50 hz clock signal (switches its value to the opposite every 0.01 seconds, meaning a rising edge every 0.02 seconds, 50 Hz):

record(calc, "$(P)$(R)Clock"){      
    field(CALC, "VAL = 1 ? 0 : 1")  
    field(SCAN, ".01 second")       
}                                   

And made a CA link to this instead of the PandA, and as expected I got things processing at 50 Hz. One thing to note is that the default regular scan rates supported by EPICS base is every 0.1 seconds. I image in your case the records are actually set to scan at I/O Interrupt level though, and the polling rate in the IOC is actually not related to the EPICS record scan rates, correct?

I don't think I'd need to go that fast, but it would be nice if we could get this to ~20 Hz let's say.

I think some kind of setup to specify poll periods for the different change groups would work for my case.

[coretl]

Some care is needed here:

PandABlocks-ioc/src/pandablocks_ioc/ioc.py

Lines 1978 to 1985 in 8e25d64

	# Its possible a bit_out field will be on for pulses shorter
	# than our polling time. To represent this we will invert the
	# value now, then set it back on the next update
	if converted_value == record_info.record.get():
	fields_to_reset.append(
	(record_info.record, converted_value)
	)
	converted_value = not converted_value

We have a requirement to see pulse widths shorter than the poll period, so if the value has changes within the poll period then we toggle it and toggle it back again on the next poll. This means that a 100Hz will toggle the TTLOUT PV as fast as it can, which means 5Hz given the 0.1s *CHANGES period.

I am also not keen to go much faster than 10Hz because this is Python after all, and if it works most of the time but occasionally stutters then that will make something fail in the middle of an experiment.

Could we increase the update rate for *CHANGES to 20Hz? That would give 10Hz updates of bit_out PVs, and I don't think we should promise any faster than that.

SOLEIL do this by sending UDP triggers directly out of the FPGA, so maybe this is an idea worth exploring further. What kind of things are you looking to process on the EPICS side?

[jwlodek]

For me, the use case is as follows:

I've written an areaDetector IOC that can read previously collected datasets either from an HDF file directly or via tiled endpoint. Once the data read back, it behaves like any other areaDetector, just with the catch that when it reaches the end of the dataset it either stops, or loops back to the front.

The main use case for this is to be able to run existing scans (or something very close to them) w/ real data, so that we can develop real-time analysis/processing software without requiring beamtime. Recently, I've upgraded this model to be able to also simulate hardware trigger support. This works by creating a CA link to a binary signal that is responsible for simulating the TTL signal. The playback IOC is smart enough to be able to be programmed to use either rising/falling edge, gated exposure, or gated acquisition. I have tested this with a simple CLOCK PV that just switches between 1 and 0 every 0.01 seconds, which produces a 50 Hz signal, and the playback runs at 50 Hz without issue.

What I am trying to do here, is basically have a setup so that we can run full panda based flyscan tests in the lab without access to the actual detector. There are plenty of PandA devices available, but it's a bit harder to have a Pilatus 2M available (and even harder to get it real x-ray data we can use for analysis). We do a lot of workflow/processing steps that happen on stop documents for example, and it'd be really nice to re-create the whole setup in the lab. I actually have this working fine now, with the catch that I get ~5 Hz as you said.

It's possible I can work around this by finding a device that offers a TTL input with a much faster polling rate available to an IOC, but I figured I'd see if this was possible at the PandA level.

[coretl]

As this is a single use case that is entirely under your control, could you open an asyn IP connection directly to the PandA TCP socket, and fire "*CHANGES.BITS?" at it as fast as you can from a thread in the playback IOC? That would skip the slow python level, and shouldn't put much extra load on the PandA TCP server...

[coretl]

Alternatively, you could do the same by connecting the playback IOC to the data connection in ASCII mode, and each time you get a new line then treat that as trigger:

$ nc 172.23.252.201 8889
NO_HEADER NO_STATUS
 0 0 0 0 0 0 6.4e-08 0.500000064 0 0 0
 0 0 0 0 0 0 1.000000064 1.500000064 0 0 0
 0 0 0 0 0 0 2.000000064 2.500000064 0 0 0
 0 0 0 0 0 0 3.000000064 3.500000064 0 0 0
 0 0 0 0 0 0 6.4e-08 0.500000064 0 0 0
 0 0 0 0 0 0 1.000000064 1.500000064 0 0 0
 0 0 0 0 0 0 2.000000064 2.500000064 0 0 0

[coretl]

That has the other benefit of letting you ensure that you have the right number of triggers, as you are guaranteed to get a new line for every PandA trigger, whereas *CHANGES could drop one (intentionally)

[jwlodek]

That should work, will do some testing, and update back here.