Release 0.1.2 #47

wilke0818 · 2024-06-02T18:19:48Z

Created Audio Pydantic model class for storing an Audio representation that can support mono and stereo audio files (Task: Internal architecture (data structures) #42)
- Audio class offers different formats for creating instances and importantly maintains a consistent internal representation of the Audio as a torch.Tensor of shape (num_channels, num_samples)
- Audios maintain, but currently do not utilize metadata information
Created AudioDataset class to help manage large number of Audios and importantly to offer functionality for running Audio tasks and pipelines with Pydra in an efficient manner
Rewrote data_augmentations and preprocessing to show use cases and code simplification that these abstract data types will allow (Task: data augmentation. #43)

…r prepping a set of Audios for a Pydra task

…ntations to show their use

codecov-commenter · 2024-06-02T18:23:37Z

Codecov Report

Attention: Patch coverage is 84.92176% with 106 lines in your changes missing coverage. Please review.

Project coverage is 65.51%. Comparing base (43f451c) to head (e95f8ec).

Files	Patch %	Lines
src/senselab/utils/data_structures/dataset.py	66.00%	34 Missing ⚠️
src/senselab/utils/data_structures/audio.py	62.06%	22 Missing ⚠️
src/senselab/utils/data_structures/video.py	60.60%	13 Missing ⚠️
src/senselab/utils/device.py	72.41%	8 Missing ⚠️
...lab/audio/tasks/speech_to_text_evaluation_pydra.py	0.00%	7 Missing ⚠️
src/senselab/audio/tasks/data_augmentation.py	68.42%	6 Missing ⚠️
src/senselab/audio/tasks/preprocessing_pydra.py	0.00%	5 Missing ⚠️
src/senselab/audio/tasks/voice_cloning.py	0.00%	4 Missing ⚠️
src/senselab/audio/tasks/speech_to_text.py	0.00%	2 Missing ⚠️
src/senselab/utils/tasks/cca_cka.py	95.55%	2 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##            main      #47       +/-   ##
==========================================
+ Coverage   9.06%   65.51%   +56.45%     
==========================================
  Files         17       37       +20     
  Lines        298      957      +659     
==========================================
+ Hits          27      627      +600     
- Misses       271      330       +59

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…selab into audio_abstract_dtype

fabiocat93 · 2024-06-03T03:50:45Z

thank you, @wilke0818; this is great! I have made some changes to align this branch with main and fixed some formatting.

Some suggestions:

please, use DeviceType (as in senselab.utils.functions; actually, we can move this to senselab.utils.device).
Also, we prefer specifying the device instead of using use_gpu (this is not a binary choice; e.g., we do support MPS, too)
I have some concerns about the AudioDataset class (do we really need it?). I feel we need to discuss this further. And if the answer is yes, can you make it a child class of Pydantic.BaseModel as you did with Audio?

Adding utility functions

fabiocat93 · 2024-06-04T22:21:01Z

src/senselab/audio/tasks/data_augmentation.py

-def augment_hf_dataset(
-    dataset: Dict[str, Any], augmentation: Compose, audio_column: str = "audio"
-) -> Dict[str, Any]:
+def augment_audio_dataset(


Can we say "audios" instead of "audio_dataset" since it's a list of audios for how it is rn?

Also, not urgent, for the very good documentation that we will write, it's a good idea to include some suggestions for parameters to use for the different types of augmentations

fabiocat93 · 2024-06-04T22:24:04Z

src/senselab/audio/tasks/data_augmentation.py

+    """
+    augmentation.output_type = "dict"
+    new_audios = []
+    device_type, dtype = _select_device_and_dtype(


how do you manage the scenario when the developer wants to use a device which is not supported? this is the question you asked me

src/senselab/audio/tasks/data_augmentation.py

src/senselab/audio/tasks/preprocessing.py

fabiocat93 · 2024-06-05T13:53:30Z

As a general comment, we have many for loops, which may be a sign that we still need to work on optimizing the code. will keep this for our discussion later today @wilke0818

fabiocat93 · 2024-06-05T13:56:08Z

src/senselab/audio/tasks/preprocessing_pydra.py

in general, if we create some workflows, it makes sense having a _pydra version of the scripts. Otherwise, we can probably remove marking functions as pydra tasks or we can do it in the same .py file. will leave this here for our discussion later today @wilke0818

fabiocat93 · 2024-06-05T14:08:03Z

src/senselab/utils/data_structures/datasets.py

+from senselab.utils.data_structures.video import Video
+
+
+class SenselabDataset(BaseModel):


as we said, we need to merge our Dataset classes @wilke0818

fabiocat93 · 2024-06-05T14:13:41Z

src/senselab/utils/data_structures/datasets.py

+        """
+        pass
+
+    def create_audio_split_for_pydra_task(self, batch_size: int = 1) -> List[List[Audio]]:


Would we also need a method for combining results together?

that's a good point. Worth considering how useful we think this Pydra split functionality is

fabiocat93 · 2024-06-05T14:15:43Z

src/senselab/utils/data_structures/video.py

+            (e.g. participant demographics, video settings, location information)
+    """
+
+    frames: torch.Tensor


Should we validate the shape of this torch.Tensor?

Yeah we can, I think it should be 4 dimensions?

fabiocat93 · 2024-06-05T14:20:14Z

src/senselab/utils/device.py

+    MPS: str = "mps"
+
+
+def _select_device_and_dtype(


should we maybe have 3 params?

requested_device (optional)

available_devices (we can compute this)

compatible_devices (this depends on the models)

in this way, we can more easily handle when the user asks for a device that is not available or compatible

fabiocat93 · 2024-06-05T14:23:26Z

src/tests/audio/tasks/data_augmentation_test.py

+            Audio(waveform=stereo_audio[0].waveform[1], sampling_rate=stereo_audio[0].sampling_rate),
+        ]
+    ).create_audio_split_for_pydra_task(2)
+    batch_inverted = augment_audio_dataset(batched_audio[0], apply_augmentation)


have you tested this code with and without GPU?

no, as I mentioned before, I am not sure exactly how we would go about realistically testing on GPU. We could mock it, but that seems like it doesn't really test

fabiocat93 · 2024-06-05T14:24:16Z

src/tests/utils/data_structures/datasets_test.py

this will need to be adjusted based on the merge we will do

Probably mostly creation logic and testing. Probably easiest to merge your branch in and just clean it up. Hopefully there'll be minimal conflicts.

… tests)

adding some more API classes (participant, session, dataset)

fabiocat93

Done in person

wilke0818 and others added 17 commits May 31, 2024 15:41

Added abstract data type for Audio and an AudioDataset that allows fo…

608495a

…r prepping a set of Audios for a Pydra task

Initiate speaker diarization

f908a40

Add working non-optimized diarization code

6ae0b7d

Add todo notes

798184c

Add buggy map to pyannote 3.1 diarization

11e0e7a

pyannote wip

6962372

functioning example of diarization with pyannote (to be cleaned)

88cea40

adding functioning example from huggingface

8083df5

Clean up diarization code

7e84f31

Remove other-row diarizations from output

ac1cab1

Add batching and fix issue with all segments in return

8e48ea7

Clean up and comment in Google style

0f7e7df

Add pydra version of pyannote diarize 3.1

25cf6fa

Update header comment

df1052c

Fix formatting and type issues

842d8c7

Add model name and revision to args

3f56932

Audio data types and reimplmentations of preprocessing and data augme…

c38cdcb

…ntations to show their use

wilke0818 requested a review from fabiocat93 June 2, 2024 18:19

wilke0818 and others added 5 commits June 2, 2024 22:43

Make fixes for automatic testing and good code practices

2f078dd

preparing for merging to main

c03fd19

Merge remote-tracking branch 'origin/main' into audio_abstract_dtype

c0ad6ac

Merge branch 'audio_abstract_dtype' of https://github.com/sensein/sen…

d549f48

…selab into audio_abstract_dtype

fixing codespell config

3e2e6b6

fabiocat93 assigned fabiocat93 and wilke0818 Jun 3, 2024

fabiocat93 added 3 commits June 3, 2024 09:42

adding speech to text evaluation task

9993456

adding cca and cka functions

62a88aa

adding cosine similarity function

97e6102

fabiocat93 and others added 3 commits June 3, 2024 19:00

treating cka kernels with enum

66ff00b

fixing style issues

3ba1620

Merge pull request #48 from sensein/utility_functions

66c646b

Adding utility functions

fabiocat93 changed the title ~~Audio abstract dtype~~ Release 0.1.2 Jun 4, 2024

This was linked to issues Jun 4, 2024

Task: Utilities #30

Closed

Task: Preprocessing #24

Closed

fabiocat93 added the enhancement New feature or request label Jun 4, 2024

fabiocat93 and others added 2 commits June 4, 2024 15:06

adding some more API classes (participant, session, dataset)

7083fac

Reorganized Audio and created more general dataset object

a27e7d0

fabiocat93 reviewed Jun 4, 2024

View reviewed changes

fabiocat93 reviewed Jun 5, 2024

View reviewed changes

src/senselab/audio/tasks/data_augmentation.py Outdated Show resolved Hide resolved

fabiocat93 reviewed Jun 5, 2024

View reviewed changes

src/senselab/audio/tasks/preprocessing.py Outdated Show resolved Hide resolved

fabiocat93 reviewed Jun 5, 2024

View reviewed changes

fabiocat93 and others added 5 commits June 5, 2024 12:32

updating dependencies and CI workflow (exploring step retry with unit…

570457b

… tests)

fixing github CI

ff3170d

Merge pull request #50 from sensein/utility_functions

29b4dfc

adding some more API classes (participant, session, dataset)

Fixing Fabio's comments from PR w/ Fabio

060c9b6

Removed function calling delted function

e95f8ec

fabiocat93 approved these changes Jun 5, 2024

View reviewed changes

fabiocat93 removed the enhancement New feature or request label Jun 5, 2024

fabiocat93 merged commit 96b7ff0 into main Jun 5, 2024
4 checks passed

fabiocat93 deleted the audio_abstract_dtype branch June 17, 2024 04:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 0.1.2 #47

Release 0.1.2 #47

wilke0818 commented Jun 2, 2024

codecov-commenter commented Jun 2, 2024 •

edited

Loading

fabiocat93 commented Jun 3, 2024

fabiocat93 Jun 4, 2024

fabiocat93 Jun 5, 2024

fabiocat93 Jun 4, 2024

wilke0818 Jun 5, 2024

fabiocat93 commented Jun 5, 2024

fabiocat93 Jun 5, 2024

fabiocat93 Jun 5, 2024

fabiocat93 Jun 5, 2024

wilke0818 Jun 5, 2024

fabiocat93 Jun 5, 2024

wilke0818 Jun 5, 2024

fabiocat93 Jun 5, 2024

wilke0818 Jun 5, 2024

fabiocat93 Jun 5, 2024

wilke0818 Jun 5, 2024

fabiocat93 Jun 5, 2024

wilke0818 Jun 5, 2024

fabiocat93 left a comment

		from senselab.utils.data_structures.video import Video


		class SenselabDataset(BaseModel):

Release 0.1.2 #47

Release 0.1.2 #47

Conversation

wilke0818 commented Jun 2, 2024

codecov-commenter commented Jun 2, 2024 • edited Loading

Codecov Report

fabiocat93 commented Jun 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabiocat93 commented Jun 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabiocat93 left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 2, 2024 •

edited

Loading