From Detection to Narration and Explanation

[arcyleung]

Hello friends, Frigate has been working amazing for my needs, and recently I built a small integration on top of it. Essentially it combines detection capability of TensorRT and another vision-text model like LLaVA to narrate and explain events as they are being recorded, so there's a text-searchable transcript.

I am gauging if there's community interest for this type of integration for multi-modal workflows (image+video) support, much like how TensorRT YOLO is currently integrated. I'm willing to put in effort to polish it further and contribute it upstream here.

You can find a video demo here and code on my repo, thanks!

[genevera]

I'm super interested in this.

[arcyleung]

Thanks for your interest! I'll work on improving the setup documentation once I get back from travels in the next few days.

[genevera]

I put in a PR that implements some missing things and fixes some broken ones 🙃