Good (and simple) way to add persistence when recording events?

[aaossa]

Hi, I'm new to using this library and has been really easy to make everything work (thanks for that 🙌).

In my use case, I need to record events in a persistent manner for later processing. As an example, imagine that I need to store all the messages sent in a Discord channel (at the moment, everything is ready except the "store" part). I was thinking that a file and pandas.DataFrame could do the trick (using PostgreSQL seems overkill), but considering the "async nature" of the library and not being thread-safe (maybe I'm wrong here) I was wondering if is there a better solution or best practice that I should consider. Is using pandas a good solution? What possible mistakes/errors should I be aware of (for example, using a threading.Lock for IO operations)?

I looked for examples but couldn't find any, but if there's a recommended approach for this I could happily push a PR with a working example (once it works for me of course). Thanks ✨!

[aaossa]

Seems that using a producer/consumer approach using asyncio.Queue (see this SO answer) is a good enough approach. My consumer would receive an event to be logged and then open, write and close the output file, while the producer would be the event handler (for example, on_message). This seems like a good idea at the moment, so I'll try to implement it. I'll leave this discussion open for a while in case some better approach appears (or this one fails).

PD: Just to be clear, I need this approach to avoid inconsistencies in the output file and access the file in a thread-safe manner

[aaossa]

In the end I used peewee to manage connections to an SqliteQ database 👌 If somebody has questions or if mantainers are interested in a PR with an example, just let me know. Kudos to the community on Discord for the help ✨