EdgeXFoundry 3.1 gets hung and unresponsive if auto events are read with 1 millisecond as interval for device-rest-go

[mohammedimrankasab]

Hi,

Currently, I am using the EdgeXFoundry 3.1 docker-compose installation with device rest and device as the only device services, and all of the auto events are disabled except for the device rest's 2way-rest-device with json resource's auto event enabled with an interval of 1 millisecond

This device rest will read the data from a simple go server with an endpoint /api/json and it will always be returning JSON response payload as {"message":"hello"}

To tackle the excess events accumulation on the Redis volume, I have also enabled the retention running every 30s to have a min of 10 and a maximum of 25 events. This cleanup using retention is also working as expected

The setup is running on the Windows machine with docker desktop setup allocation of 1cpu and 4GB of memory.

We observe that EdgeXFoundry is becoming slow and eventually unresponsive with the application crashing within a few minutes. Tried to debug through the logs but as containers hung up, logs were not available.

Kindly help here and let us know if the setup requires any higher configuration w.r.t resources or any other tweaks to be made.

[jiekechoo]

Can you try to use the command docker compose up in the terminal and paste all logs here?

[cloudxxx8]

You disable the Core Data persistence to improve this situation.
However, 1 ms interval is too short to process events on your 1 CPU machine.
Try to use docker stats to check the memory and CPU usage. If the interval is lower than the machine process time, the memory will pile up until out of memory.
You need to adjust the interval to the number could work fine on your machine.

[mohammedimrankasab]

Thank you for responding quickly. As suggested I will try to tweak the processor count and the interval value.
The use case that we are trying to perform is that some initial performance benchmarking runs to evaluate how many events the EdgeXFoundry can handle by consuming messages from a REST device.
We would be happy to learn if there are better ways to perform performance benchmarking like how many events are generated per second.

Thanks again

[mohammedimrankasab]

Also, we have created an MQTT sink through eKuiper engine, and are trying to count the messages consumed by the MQTT sink per second, is this the right approach to perform benchmarking on an eKuiper sink? It would be great if you could share some stats around performance benchmarking if there any available.