Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task: Add quality check for initial vacant properties dataset #855

Closed
6 tasks
nlebovits opened this issue Aug 2, 2024 · 3 comments
Closed
6 tasks

Task: Add quality check for initial vacant properties dataset #855

nlebovits opened this issue Aug 2, 2024 · 3 comments
Assignees

Comments

@nlebovits
Copy link
Collaborator

nlebovits commented Aug 2, 2024

Describe the task

Implement functionality to check that the initial dataset of vacant properties is no more than 5% smaller than the previous run or, if there was no previous run, no smaller than 30,000 records. If this condition is not met, send an alert via Slack and email. This will involve modifying the script to query the previous record count from the PostgreSQL database and compare it with the current record count. Changes will need to be made to ./data/src/script.py and ./data/src/classes/diff_report.py.

Acceptance Criteria

  • Add a method to DiffReport class in data/src/classes/diff_report.py to get the previous record count from the PostgreSQL database.
  • Modify the main script in data/src/script.py to use the DiffReport method to get the previous record count.
  • Implement a comparison of the current and previous record counts in the main script.
  • If the current count is more than 5% smaller than the previous count or if the initial count is smaller thna 30,000 records, break and:
    • Send a Slack alert.
    • Optionally, send an email alert.

Additional context

  • The Slack alert should use the Slack API and be configured to post to a specific channel.
  • The email alert should use the SMTP library and be sent to a configured email address.
  • Ensure proper error handling and logging for both Slack and email alerts.
@nlebovits nlebovits added help wanted Extra attention is needed python high priority labels Aug 2, 2024
@nlebovits nlebovits moved this to To Do in Clean & Green Philly Aug 2, 2024
@nlebovits nlebovits moved this from To Do to In Development in Clean & Green Philly Aug 2, 2024
@nlebovits nlebovits self-assigned this Aug 2, 2024
@zigouras
Copy link
Collaborator

zigouras commented Aug 4, 2024

related to #848. Any quality failure should raise an exception and get caught and reported to Slack.

Copy link

github-actions bot commented Oct 1, 2024

This issue has been marked as stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the stale label Oct 1, 2024
@nlebovits
Copy link
Collaborator Author

effectively a duplicate of #848

@nlebovits nlebovits closed this as not planned Won't fix, can't repro, duplicate, stale Oct 3, 2024
@github-project-automation github-project-automation bot moved this from In Development to Live in Clean & Green Philly Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

2 participants