-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add fail_on_file_not_exist option to SFTPToS3Operator #44320
Conversation
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
|
providers/src/airflow/providers/amazon/aws/transfers/sftp_to_s3.py
Outdated
Show resolved
Hide resolved
ae25482
to
2984be7
Compare
providers/src/airflow/providers/amazon/aws/transfers/sftp_to_s3.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Ephraim Anierobi <[email protected]>
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
* Add fail_on_file_not_exist option to SFTPToS3Operator * Fixed test for fail_on_file_not_exist option. * �Remove else condition with fail_on_file_not exist option Co-authored-by: Ephraim Anierobi <[email protected]> --------- Co-authored-by: Ephraim Anierobi <[email protected]>
Fixes: #40576
I added
fail_on_file_not_exist
param toSFTPToS3Operator
so that user can configure the parameter and operator will not fail in case of sftp file not exist.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.Here's the test result ran in my local environment.
you can see the dag does not fail with
fail_on_file_not_exist
but still shows logs that the file does not exist.sftp_to_s3_dag.py