Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS Glue Job Operator Not Able to Read Jinja Format #45036

Open
2 tasks done
nicholascz666666 opened this issue Dec 18, 2024 · 6 comments
Open
2 tasks done

AWS Glue Job Operator Not Able to Read Jinja Format #45036

nicholascz666666 opened this issue Dec 18, 2024 · 6 comments
Assignees
Labels

Comments

@nicholascz666666
Copy link

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

2.10.1

Apache Airflow version

9.1.0

Operating System

N/A

Deployment

Amazon (AWS) MWAA

Deployment details

I tried to configure the Number Of Workers and worker type within the run_job_kwargs parameter with jinja format like the following
run_job_kwargs={"NumberOfWorkers": '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}', "WorkerType":"dummy_type"}

What happened

the value with the jinja format was not recognized during run time and it shows the error that the Number Of Workers should have input type int rather than a string '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}'

What you think should happen instead

'{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}' should pick up the real value during run time

How to reproduce

this could be reproduced by creating a dummy function to push some value to the xcom first then using a glue job operator with specifying
run_job_kwargs={"NumberOfWorkers": '{{ ti.xcom_pull(task_ids="dummy",key="dummy") }}', "WorkerType":"dummy_type"}

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@nicholascz666666 nicholascz666666 added area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Dec 18, 2024
Copy link

boring-cyborg bot commented Dec 18, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@dosubot dosubot bot added the provider:amazon-aws AWS/Amazon - related issues label Dec 18, 2024
@jayceslesar
Copy link
Contributor

Does this persist even if you have render_template_as_native_obj=True to the DAG constructor?

@gopidesupavan
Copy link
Member

Yep, this field is not templated. You may raise changes for this, think this is similar to https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/amazon/aws/operators/glue.py#L89 create_job_kwargs

@gopidesupavan gopidesupavan added good first issue and removed kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Dec 22, 2024
@basvi-chunara
Copy link

I would like to work on this. Can this issue be assigned to me?

@potiuk
Copy link
Member

potiuk commented Dec 23, 2024

Coordinate with @nicholascz666666 @basvi-chunara becasue he was assigned before - but I assigned both of you now.

@eladkal eladkal added the kind:feature Feature Requests label Dec 23, 2024
@basvi-chunara
Copy link

okay, thank you for letting me know

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants