Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/test deployed dags #481

Merged
merged 23 commits into from
Oct 10, 2023
Merged

Feature/test deployed dags #481

merged 23 commits into from
Oct 10, 2023

Conversation

ggsdc
Copy link
Member

@ggsdc ggsdc commented Oct 9, 2023

Added a DAG that automatically runs all examples uploaded (one per dag) and runs them against cornflow to check that everything works properly.
Some small changes for DAGs to run properly.
Finished SendGrid integration in Airflow.

  • Add new unit test to improve coverage.
  • Prepare PR in cornflowdags private repo with these changes.

Closes #475

@codecov
Copy link

codecov bot commented Oct 9, 2023

Codecov Report

Merging #481 (1b2ab68) into development (b0f0195) will increase coverage by 0.17%.
The diff coverage is 56.07%.

@@               Coverage Diff               @@
##           development     #481      +/-   ##
===============================================
+ Coverage        78.77%   78.94%   +0.17%     
===============================================
  Files              234      263      +29     
  Lines            12784    14807    +2023     
===============================================
+ Hits             10070    11689    +1619     
- Misses            2714     3118     +404     
Flag Coverage Δ
client-tests 77.84% <ø> (-0.42%) ⬇️
server-tests 82.94% <97.22%> (-0.39%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
cornflow-dags/DAG/activate_dags.py 91.30% <100.00%> (+0.39%) ⬆️
cornflow-dags/DAG/bar_cutting/__init__.py 100.00% <ø> (ø)
cornflow-dags/DAG/facility_location/__init__.py 100.00% <100.00%> (ø)
...imension_bin_packing/solvers/right_corner_model.py 96.24% <100.00%> (ø)
cornflow-dags/DAG/update_all_schemas.py 77.64% <100.00%> (ø)
cornflow-dags/DAG/update_dag_registry.py 43.33% <100.00%> (-1.83%) ⬇️
cornflow-server/cornflow/cli/schemas.py 97.77% <ø> (ø)
cornflow-server/cornflow/endpoints/execution.py 74.82% <100.00%> (+4.38%) ⬆️
cornflow-server/cornflow/models/dag.py 88.88% <ø> (+1.08%) ⬆️
cornflow-server/cornflow/models/execution.py 100.00% <100.00%> (+7.50%) ⬆️
... and 5 more

... and 79 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Comment on lines +31 to +32
notify = getattr(app, "notify", True)
if not notify:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change @AlejandraGalan done over your code is to make sure that this change is compatible with older versions cornflow-client just in case


instance_id = response["id"]

config = {"timeLimit": 60, "msg": True, "seconds": 60}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marioncottard I am getting a lot of weird behaviours with this part of timeLimit and seconds. Could you start preparing a PR that modifies all example DAGs to use the translation dictionaries that we had created?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay

Comment on lines +189 to +191
new_config, config_errors = json_schema_extend_and_validate_as_string(
config_schema, kwargs["config"]
)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marioncottard This is the new validation that adds the default values (if needed) before validating the schema

@ggsdc ggsdc linked an issue Oct 10, 2023 that may be closed by this pull request
Copy link
Collaborator

@marioncottard marioncottard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

Deleted unreacheble code on executions
@ggsdc ggsdc merged commit 8590599 into development Oct 10, 2023
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Testing DAG
2 participants