Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate the audio modality in CoCa #94

Draft
wants to merge 129 commits into
base: main
Choose a base branch
from
Draft

Commits on May 7, 2024

  1. feat: add basic webdataset

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d909ae0 View commit details
    Browse the repository at this point in the history
  2. fix: dim of cls token

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    e233676 View commit details
    Browse the repository at this point in the history
  3. feat: simple console logging

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    9986691 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c47b6c1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    70823e1 View commit details
    Browse the repository at this point in the history
  6. fix: register nce loss

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    0c87d91 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    b652a7d View commit details
    Browse the repository at this point in the history
  8. chore: add config

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b0e933a View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d8d5a5f View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    4ea65c8 View commit details
    Browse the repository at this point in the history
  11. chore: update coca config

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    dfe88c9 View commit details
    Browse the repository at this point in the history
  12. fix: rebase

    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    5a3e844 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Configuration menu
    Copy the full SHA
    043384d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3cd9244 View commit details
    Browse the repository at this point in the history
  3. fix: cuda env and run script

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b09d20e View commit details
    Browse the repository at this point in the history
  4. chore: update coca config

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    e09745d View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5a74dee View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    f7b725c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c7308e2 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    32d0b19 View commit details
    Browse the repository at this point in the history
  9. fix: webdataset slow loading

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    55c039f View commit details
    Browse the repository at this point in the history
  10. fix: add batching

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    966d237 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    dacf639 View commit details
    Browse the repository at this point in the history
  12. fix: webloader

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b40ecd5 View commit details
    Browse the repository at this point in the history
  13. fix: dataset factory

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a9ce132 View commit details
    Browse the repository at this point in the history
  14. fix: webdataset

    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    63ef47c View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. fix: loss accumulation

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    d6d84dc View commit details
    Browse the repository at this point in the history
  2. fix: loss average for eval

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    3d04f78 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e65a3cd View commit details
    Browse the repository at this point in the history
  4. fix: coca collator

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    622570d View commit details
    Browse the repository at this point in the history
  5. fix: loss normalization

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    efedc77 View commit details
    Browse the repository at this point in the history
  6. feat: add clip loss

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    eba23a9 View commit details
    Browse the repository at this point in the history
  7. fix: use clip loss in coca

    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    9bdc830 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b3aff91 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    dffe644 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. fix: gradient accumulation

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    d8bdebc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    244fac9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0524acd View commit details
    Browse the repository at this point in the history
  4. fix: log weighted loss

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    df48b62 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2945a03 View commit details
    Browse the repository at this point in the history
  6. fix: loss logging

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    2d0e3f3 View commit details
    Browse the repository at this point in the history
  7. feat: add local clip loss

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1c0993f View commit details
    Browse the repository at this point in the history
  8. fix: clip loss

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    0e3b239 View commit details
    Browse the repository at this point in the history
  9. fix: add barrier to eval

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    7459ee0 View commit details
    Browse the repository at this point in the history
  10. feat: print global batch size

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1c0f456 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    a04bf97 View commit details
    Browse the repository at this point in the history
  12. fix: validation set loading

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    9086af6 View commit details
    Browse the repository at this point in the history
  13. fix: webdataset splitter

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    351990a View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    c90df9d View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    860b8fd View commit details
    Browse the repository at this point in the history
  16. refactor: evaluator

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    962c4ca View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    2d1ea92 View commit details
    Browse the repository at this point in the history
  18. feat: add wandb grouping

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1fb4fba View commit details
    Browse the repository at this point in the history
  19. chore: update coca config

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    22c77cd View commit details
    Browse the repository at this point in the history
  20. fix: coca config

    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    6d980e7 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Configuration menu
    Copy the full SHA
    dfb7884 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    da27f0d View commit details
    Browse the repository at this point in the history
  3. fix: webdataset

    spravil committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    016bed3 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. feat: add conformer audio encoder

    manasMauryax authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    587c52b View commit details
    Browse the repository at this point in the history
  2. feat: make CoCa audio compatible

    manasMauryax authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    1ba15bc View commit details
    Browse the repository at this point in the history
  3. test: change config and dummy dataset for E2E CoCa test

    manasMauryax authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    d71cbdb View commit details
    Browse the repository at this point in the history
  4. feat: add new Dataset class for Audio/Vision

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    e824400 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    fcaccea View commit details
    Browse the repository at this point in the history
  6. fix: to avoid torch.tensor(tensor)

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    5dd535e View commit details
    Browse the repository at this point in the history
  7. fix: add argument to ignore padding indices

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    302d396 View commit details
    Browse the repository at this point in the history
  8. test: uptate tests to comply with changes

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    94c85ce View commit details
    Browse the repository at this point in the history
  9. feat: add entrypoint for audio coca tokenizer

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    ceaa897 View commit details
    Browse the repository at this point in the history
  10. chore: add configs

    These can help run audio-only, vision-only or audio-vision experiments!
    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b3b7f77 View commit details
    Browse the repository at this point in the history
  11. chore: update pyproject.toml

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    85c0c69 View commit details
    Browse the repository at this point in the history
  12. feat: allow masking of "pad" keys

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    e4c956c View commit details
    Browse the repository at this point in the history
  13. feat: implement Conformer from scratch

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    94e042b View commit details
    Browse the repository at this point in the history
  14. test: fix to comply to changes

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    c6c6f30 View commit details
    Browse the repository at this point in the history
  15. test: remove deprecated test

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    03db6ec View commit details
    Browse the repository at this point in the history
  16. feat: improve arrow dataset

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    9704c09 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    7c5e063 View commit details
    Browse the repository at this point in the history
  18. chore: fix configs to comply to changes

    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    a554b82 View commit details
    Browse the repository at this point in the history
  19. chore: add script which generates tokenizer

    On the basis of a training.txt file and number
    of assimilation operations, a bpecodes file is
    generated which is used to create bpe_to_ind
    and ind_to_bpe dictionary pickles required
    for tokenization and detokenization.
    mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    75a0ebe View commit details
    Browse the repository at this point in the history
  20. fix: accelerate import

    thomaschhh authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    50add61 View commit details
    Browse the repository at this point in the history
  21. refactor: introduce global constants

    thomaschhh authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    433ac9d View commit details
    Browse the repository at this point in the history
  22. fix: constant renaming

    thomaschhh authored and mmaurya committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    aa50864 View commit details
    Browse the repository at this point in the history
  23. fix: web dataset integration

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    faeec41 View commit details
    Browse the repository at this point in the history
  24. chore: add todo statement

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    1c6fcff View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    e90a8d5 View commit details
    Browse the repository at this point in the history
  26. chore: update start script

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    42f469b View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    03082b8 View commit details
    Browse the repository at this point in the history
  28. fix: coca webdataset config

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    78efe14 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    deb0788 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    4a9695d View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    f04243a View commit details
    Browse the repository at this point in the history
  32. feat: add video transforms

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    cad8d11 View commit details
    Browse the repository at this point in the history
  33. chore: add video config

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    3653922 View commit details
    Browse the repository at this point in the history
  34. fix: video coca

    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    3b72cd2 View commit details
    Browse the repository at this point in the history
  35. chore: add audio config

    thomaschhh committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    357c76e View commit details
    Browse the repository at this point in the history
  36. feat: add audio transform

    thomaschhh committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    2a4f45d View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    ae71bb7 View commit details
    Browse the repository at this point in the history
  38. fix: collator

    thomaschhh committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b04652a View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    137d4f2 View commit details
    Browse the repository at this point in the history
  40. chore: adjust forward call

    thomaschhh committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    d93e6e5 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    9528bb6 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. feat: add conformer audio encoder

    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0c0317a View commit details
    Browse the repository at this point in the history
  2. feat: make CoCa audio compatible

    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    5f63246 View commit details
    Browse the repository at this point in the history
  3. test: change config and dummy dataset for E2E CoCa test

    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    857867e View commit details
    Browse the repository at this point in the history
  4. feat: add new Dataset class for Audio/Vision

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c9ad825 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    97ecb20 View commit details
    Browse the repository at this point in the history
  6. fix: to avoid torch.tensor(tensor)

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    4463ae7 View commit details
    Browse the repository at this point in the history
  7. fix: add argument to ignore padding indices

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b410073 View commit details
    Browse the repository at this point in the history
  8. test: uptate tests to comply with changes

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    6c0ced6 View commit details
    Browse the repository at this point in the history
  9. feat: add entrypoint for audio coca tokenizer

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    e0577b7 View commit details
    Browse the repository at this point in the history
  10. chore: add configs

    These can help run audio-only, vision-only or audio-vision experiments!
    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f40f7f6 View commit details
    Browse the repository at this point in the history
  11. chore: update pyproject.toml

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    8bed2c2 View commit details
    Browse the repository at this point in the history
  12. feat: allow masking of "pad" keys

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    743d999 View commit details
    Browse the repository at this point in the history
  13. feat: implement Conformer from scratch

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    b2cb27c View commit details
    Browse the repository at this point in the history
  14. test: fix to comply to changes

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    7a23ced View commit details
    Browse the repository at this point in the history
  15. test: remove deprecated test

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    363701c View commit details
    Browse the repository at this point in the history
  16. feat: improve arrow dataset

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    d403c62 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    bc0d9f0 View commit details
    Browse the repository at this point in the history
  18. chore: fix configs to comply to changes

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    bae2f0f View commit details
    Browse the repository at this point in the history
  19. chore: add script which generates tokenizer

    On the basis of a training.txt file and number
    of assimilation operations, a bpecodes file is
    generated which is used to create bpe_to_ind
    and ind_to_bpe dictionary pickles required
    for tokenization and detokenization.
    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0cf2290 View commit details
    Browse the repository at this point in the history
  20. fix: accelerate import

    thomaschhh authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f871403 View commit details
    Browse the repository at this point in the history
  21. refactor: introduce global constants

    thomaschhh authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    c9f58b9 View commit details
    Browse the repository at this point in the history
  22. fix: constant renaming

    thomaschhh authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1806062 View commit details
    Browse the repository at this point in the history
  23. fix: disable mamba imports

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    f8181d0 View commit details
    Browse the repository at this point in the history
  24. chore: update audio coca arrow dataset config

    mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    25803ca View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Configuration menu
    Copy the full SHA
    b27bf78 View commit details
    Browse the repository at this point in the history
  2. feat: add audio transform

    thomaschhh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    cca934e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bcc7e9d View commit details
    Browse the repository at this point in the history
  4. fix: prepare_sample

    thomaschhh committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    e78b2ea View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8d5f459 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    162c965 View commit details
    Browse the repository at this point in the history