Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: SummarizationAccuracyMetrics transform to handle multiple target outputs more efficiently #317

Merged
merged 41 commits into from
Aug 8, 2024
Merged

refactor: SummarizationAccuracyMetrics transform to handle multiple target outputs more efficiently #317

merged 41 commits into from
Aug 8, 2024

Commits on Jul 3, 2024

  1. Added metric to factual knowledge + unit/integration tests 

    cr: https://code.amazon.com/reviews/CR-135854933
    @kirupang-code
    kirupang-code committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    6d66b62 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. fixed changes from PR comments

    @kirupang-code
    kirupang-code committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    cc866ed View commit details
    Browse the repository at this point in the history

  2. Deleted metrics.py and restored code in util.py

    @kirupang-code
    kirupang-code committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    843d9f6 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. added factual knowledge metrics to constants.py

    @kirupang-code
    kirupang-code committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    c2f9efb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of github.com:aws/fmeval

    @kirupang-code
    kirupang-code committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    d7e5fa5 View commit details
    Browse the repository at this point in the history

  3. added factual knowledge metrics to be included in binary score

    @kirupang-code
    kirupang-code committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    8d9bf4f View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. updated score descriptions for factual knowledge

    @kirupang-code
    kirupang-code committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    1aee116 View commit details
    Browse the repository at this point in the history

  2. feat: add configurable param logical_operator (OR/AND) to factual kno… 

    …weldge
    @kirupang-code
    kirupang-code committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    d8c29da View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of github.com:aws/fmeval

    @kirupang-code
    kirupang-code committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    8715749 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Merge branch 'main' of github.com:aws/fmeval

    @kirupang-code
    kirupang-code committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    0ca7c47 View commit details
    Browse the repository at this point in the history

  2. fixed changes from PR comments

    @kirupang-code
    kirupang-code committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    ba43b92 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. added warning and fixed typo

    @kirupang-code
    kirupang-code committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    5e7bb50 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into main

    @kirupang-code
    kirupang-code authored Jul 12, 2024
    Configuration menu
    Copy the full SHA
    411e0fa View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. modified warnings and fixed invalid config tests for factual_knowledge

    @kirupang-code
    kirupang-code committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f1f9792 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of github.com:aws/fmeval

    @kirupang-code
    kirupang-code committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    43a9a48 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' of github.com:kirupang-code/fmeval

    @kirupang-code
    kirupang-code committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5f2d1d4 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' of github.com:kirupang-code/fmeval

    @kirupang-code
    kirupang-code committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b684515 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' of github.com:kirupang-code/fmeval

    @kirupang-code
    kirupang-code committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    e13bd5c View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. Merge branch 'main' of github.com:aws/fmeval

    @kirupang-code
    kirupang-code committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    d51f3ff View commit details
    Browse the repository at this point in the history

  2. feat: Adding BERTScore to QAAccuracy + QAAccuracySemanticRobustness

    @kirupang-code
    kirupang-code committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    2a165cf View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. fix: documentation and tests for qa accuracy + qa accuracy semantic r… 

    …obustness
    @kirupang-code
    kirupang-code committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    634e85f View commit details
    Browse the repository at this point in the history

  2. fix: lint checks

    @kirupang-code
    kirupang-code committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    c019be5 View commit details
    Browse the repository at this point in the history

  3. fix: created dataset for qa_accuracy, reverted to js_model_runner

    @kirupang-code
    kirupang-code committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    0988ac7 View commit details
    Browse the repository at this point in the history

  4. fix: integration tests by adding approx for BertScore

    @kirupang-code
    kirupang-code committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    28b449e View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. fix: moved BertScoreWithDelimiter to qa_accuracy and updated tests

    @kirupang-code
    kirupang-code committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    5b99691 View commit details
    Browse the repository at this point in the history

  2. fix: restored qa_accuracy_semantic_robustness

    @kirupang-code
    kirupang-code committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e6c6f33 View commit details
    Browse the repository at this point in the history

  3. fix: smaller dataset for integ tests to reduce runtime

    @kirupang-code
    kirupang-code committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    e3d02d6 View commit details
    Browse the repository at this point in the history

  4. fix: smaller dataset for integ tests to reduce runtime

    @kirupang-code
    kirupang-code committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    7052ef2 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' of github.com:kirupang-code/fmeval

    @kirupang-code
    kirupang-code committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4859531 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Add BertScoreMax transform for qa_accuracy

    @kirupang-code
    kirupang-code committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    fdb99b1 View commit details
    Browse the repository at this point in the history

  2. fix: lint checks

    @kirupang-code
    kirupang-code committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    cf6448a View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. fix: cleaning up code and checking reporting folder for changes

    @kirupang-code
    kirupang-code committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    c9f565d View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. fix: refactored SummarizationAccuracyMetric

    @kirupang-code
    kirupang-code committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    a84aa5b View commit details
    Browse the repository at this point in the history

  2. fix: deleted dataset file from previous PR

    @kirupang-code
    kirupang-code committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    7589756 View commit details
    Browse the repository at this point in the history

  3. refactor: added target_output_keys_provider to SummarizationAccuracyM… 

    …etric
    @kirupang-code
    kirupang-code committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    f58046a View commit details
    Browse the repository at this point in the history

  4. edited description of test function

    @kirupang-code
    kirupang-code committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    bfb7f20 View commit details
    Browse the repository at this point in the history

  5. updated assert description for target_output_keys

    @kirupang-code
    kirupang-code committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    8431857 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. fixed comments from PR

    @kirupang-code
    kirupang-code committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    410b0fc View commit details
    Browse the repository at this point in the history

  2. fixed errors in test from previous run

    @kirupang-code
    kirupang-code committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    b59547e View commit details
    Browse the repository at this point in the history

  3. add unit test and fixed type issues

    @kirupang-code
    kirupang-code committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    f9e31da View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. fix: Mocked BertScore in unit test

    @kirupang-code
    kirupang-code committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    48c06dd View commit details
    Browse the repository at this point in the history