Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add almanack score for simplified understanding through single metric #208

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

d33bs
Copy link
Member

@d33bs d33bs commented Dec 16, 2024

Description

This PR adds a "almanack score" metric which summarizes the findings of other metrics in one single value. This score is calculated using boolean metrics and a "sustainability direction" value per metric. The values are summed and divided by the number of metrics to then provide a score. This score is intended to prepare for a more statistically profound measure on repository sustainability which is formed through modeling and other related efforts.

Along the journey towards this PR I noticed and fixed the following:

  • Metrics types and type check tests for the metrics table (which presented challenges earlier and benefited the work in this PR).

References #135
Closes #207

What is the nature of your change?

  • Content additions or updates (adds or updates content)
  • Bug fix (fixes an issue).
  • Enhancement (adds functionality).
  • Breaking change (these changes would cause existing functionality to not work as expected).

Checklist

Please ensure that all boxes are checked before indicating that this pull request is ready for review.

  • I have read the CONTRIBUTING.md guidelines.
  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own contributions.
  • I have commented my content, particularly in hard-to-understand areas.
  • I have made corresponding changes to related documentation (outside of book content).
  • My changes generate no new warnings.
  • New and existing tests pass locally with my changes.
  • I have added tests that prove my additions are effective or that my feature works.
  • I have deleted all non-relevant text in this pull request template.

@d33bs d33bs marked this pull request as ready for review December 16, 2024 12:58
Copy link
Member

@gwaybio gwaybio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several discussion points this time

src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/metrics.yml Outdated Show resolved Hide resolved
src/almanack/metrics/metrics.yml Outdated Show resolved Hide resolved
@d33bs d33bs requested a review from gwaybio December 30, 2024 15:15
@d33bs
Copy link
Member Author

d33bs commented Dec 30, 2024

Thanks @gwaybio for the earlier review! I've restructured the approach here with an eye towards simplification and better documentation. When there's a chance, could I ask for another look? Thank you!

src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
src/almanack/metrics/data.py Outdated Show resolved Hide resolved
tests/metrics/test_data.py Outdated Show resolved Hide resolved
@d33bs d33bs changed the title Add sustainability score for simplified understanding through single metric Add almanack score for simplified understanding through single metric Jan 3, 2025
@d33bs
Copy link
Member Author

d33bs commented Jan 3, 2025

Thanks @gwaybio for the further review! After making some changes based on your comments here and our discussion, I wanted to make note that I've changed the language a bit within the code and in the PR description to label this score as an "almanack score" which I feel might be distinct from the "sustainability score" (or at least may need consolidation at some point in the future to better encompass the full gamut of Almanack metrics). The PR no longer closes the sustainability score issue (#135) to instead continue exploring this as we develop the Almanack.

@d33bs d33bs requested a review from gwaybio January 3, 2025 22:15
Copy link
Member

@gwaybio gwaybio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great discussions in this PR! I think the project will benefit from them.

@@ -84,6 +83,16 @@ def get_table(repo_path: str) -> List[Dict[str, Any]]:
for metric in metrics_table
]

# calculate sustainability score (modify placeholder)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# calculate sustainability score (modify placeholder)
# calculate almanack score (the function modifies the placeholder)

Comment on lines +1203 to +1204
Computes an Almanack score by normalizing boolean Almanack
table metrics in order to summarize analysis.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Computes an Almanack score by normalizing boolean Almanack
table metrics in order to summarize analysis.
Computes an Almanack score by counting boolean Almanack
table metrics to provide a quick summary of software sustainability.


Returns:
float:
The computed almanack score, normalized between 0 and 1.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still true?

Comment on lines +86 to +91
Dictionary which includes number of almanack boolean metrics
with a non-zero sustainability correction which passed, number
of almanack checks in total, and a score which is the number
of passed checks divided by the total number of checks that
represents how likely a repository will be maintained over
time based on the existing Almanack metrics.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Dictionary which includes number of almanack boolean metrics
with a non-zero sustainability correction which passed, number
of almanack checks in total, and a score which is the number
of passed checks divided by the total number of checks that
represents how likely a repository will be maintained over
time based on the existing Almanack metrics.
Dictionary of length three, including the following: 1) number of
Almanack boolean metrics that passed (numerator), 2) number of
total Almanack boolean metrics considered (denominator), and 3) a
score that represents how likely the repository will be maintained
over time based (numerator / denominator).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add test for metrics data table types based on metrics.yml types
2 participants