Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Add Eurostat source and Prefect tasks #1041

Merged
merged 49 commits into from
Oct 9, 2024
Merged

✨ Add Eurostat source and Prefect tasks #1041

merged 49 commits into from
Oct 9, 2024

Conversation

adrian-wojcik
Copy link
Contributor

@adrian-wojcik adrian-wojcik commented Sep 19, 2024

Summary

Adding Eurostat connector

Importance

To use Eurostat in Prefect 2.0

Checklist

This PR:

  • follows the guidelines laid out in CONTRIBUTING.md
  • links relevant issue(s)
  • adds/updates tests (if appropriate)
  • adds/updates docstrings (if appropriate)
  • adds an entry in CHANGELOG.md

Copy link
Contributor

@trymzet trymzet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added my comments

@trymzet trymzet changed the title Eurostat 2.0 ✨ Add Eurostat source and Prefect tasks Sep 27, 2024
Copy link
Contributor

@trymzet trymzet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added my feedback. It might be easier to create one issue to address the changes for which I haven't added suggestions (except unit tests as these should be fixed before merging).

Copy link
Contributor

@trymzet trymzet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Looks like the unit tests still don't mock the API calls.
  2. It seems docs are also not added. See https://viadot.docs.dyvenia.com/developer_guide/creating_a_source/#adding-docs and https://viadot.docs.dyvenia.com/developer_guide/creating_a_prefect_flow/#adding-docs
  3. There's one open conversation left from before

@trymzet trymzet added the 2.0 viadot 2.0 label Oct 4, 2024
Copy link

gitguardian bot commented Oct 4, 2024

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
6616985 Triggered Generic Password c3589a1 tests/unit/test_business_core.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@Rafalz13 Rafalz13 requested a review from trymzet October 7, 2024 08:42
@trymzet
Copy link
Contributor

trymzet commented Oct 8, 2024

@Rafalz13 the other "unit" tests are also not unit tests; they try to connect to the actual API. So the mocking in general is incorrect and needs fixing (once you fix it you might as well re-add the to_df() test since it used the same mocking logic as other tests?)

@Rafalz13
Copy link
Contributor

Rafalz13 commented Oct 9, 2024

@zawadzki Tests don't connect to the source after changes. Test coverage is 80%.

@Rafalz13 Rafalz13 requested a review from trymzet October 9, 2024 09:13
@trymzet trymzet merged commit c281785 into 2.0 Oct 9, 2024
3 checks passed
@trymzet trymzet deleted the eurostat_2.0 branch October 9, 2024 10:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.0 viadot 2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants