Visualize human vs machine classifications in time & space

[valentina-s]

Visualize human vs machine classifications in time & space

Take data outputs from ML models and/or human detection from the Orcasound app and visualize them to provide bioacoustic insights.

Many biological sounds made by marine organisms are correlated with important temporal cycles in the ocean, e.g. tides, daylight, moonlight, seasons, decadal trends. As Orcasound continues to generate human detections over the years, and as more ML models classify sounds in the audio streams, we’d like to visualize their patterns. We’re most interested in how the soundscape changes over time, but you could begin to look for spatial patterns, too (across the three locations that Orcasound currently maintains and the 5 that we expect to have deployed by the end of 2020). See the “Data visualization ideas” list in Orcasound’s Trello board of “bioacoustic data analysis approaches".

Required skills: Elixir, Javascript

Bonus skills: time-series analysis

Possible mentors: Scott, Val, Jesse

References and open-source building blocks:

• Associated Orcasound feature card (Trello)
  Good visualization ideas from the Santa Barbara Channel hydrophone (real-time ML outputs)
• Emerging sources of real-time and archived data that could be visualized
  • Human detections, real-time & archived for ~18 months, from the Orcasound web app (in a growing Postgres database on Heroku)
  • Machine detections, real-time & archived since Sep 2020, from the real-time inference system developed by Microsoft volunteers (raw predictions and annotated candidates in a growing Cosmo database on Azure)
  • Expert-confirmed "acoustic bouts" with orca or humpback signals that bracket human and/or machine detections

[DhananjayPurohit]

Based on the References and open-source building blocks mentioned in the project, I had developed a Visualization Dashboard through React ( https://dhananjaypurohit.github.io/OrcaDashboard/ ). I had used Google Sheets (OrcaData) to hold the same Automated detection data of a week given on the website and had implemented some visualizations on the same data.
Link to my repo (https://github.com/DhananjayPurohit/OrcaDashboard).
I would love to hear improvements or new ideas on the same.

[DhananjayPurohit]

Here is the link to my proposal.
https://docs.google.com/document/d/1bzl17zdGB56mvg0dhF-HBwWjJ2XdM9advvOKWKZV0pE/edit?usp=sharing
I would love to hear improvements or feedback on the same.

[ghost]

Hey @scottveirs! This seemed to be an interesting project within my limited capabilities. I’m looking to contribute towards this as it was an intriguing project (i.e. I don’t want to do this for google summer of code or any other purpose). I would love to collaborate with you regarding the same. If I am allowed to contribute please comment here (I’m new to open source). I’d love to share a roadmap with you before I begin and would love to be mentored by your team. Moreover if this goes well, I’d like to be a part of the Orcasound team on a much more permanent basis too. Looking forward to hearing from you!

Cheers
Harsith

[scottveirs]

Add list of emerging sources of species presence inferred from acoustic signals.

[scottveirs]

An April 2021 note from Akash regarding this GSoC topic:

On this - everything is already on Github GitHub - orcasound/aifororcas-livesystem, just not documented to a complete level, although it seems like good progress was made in March (see the open PRs). Building over what exists is possible for 2021 itself. Might be a bit messy but folks are most welcome to bite the bullet sooner.

• Training is under /fastai and deploying to Docker on Azure is under /inferencesystem and scoring models under /evaluation. Aside from me - Aayush and Prakruti are familiar with this
• The main website is under /ModeratorFrontEnd in C# & blazor - Mike has the best idea (I believe blazor is a pioneering webassembly technology which is a big enough deal for people to be interested in it for the webassembly part)
• Mike (and maybe Herman or Michelle) - have an idea of CosmosDB and API service in /ModeratorFrontEnd/ModeratorCandidates
• Automated notifications is under /NotificationSystem - Michelle (and Adele) have the best idea here
• For any misc overall or Azure stuff - Prakruti

FYI - (this is a closed source tool within Azure/MFST ecosystem) - i've seen people use Data Visualization | Microsoft Power BI to connect with existing resources and quickly spin up some dashboards.