Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes decorator #21077

Closed
wants to merge 95 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
a1c532e
[16185] Added LocalKubernetesExecutor to breeze supported executors
subkanthi Nov 22, 2021
6cbb506
Merge branch 'master' of github.com:subkanthi/airflow
subkanthi Nov 22, 2021
d16a343
Revert "[16185] Added LocalKubernetesExecutor to breeze supported exe…
subkanthi Nov 22, 2021
a336e17
Merge branch 'apache:main' into master
subkanthi Nov 30, 2021
382eedf
Merge branch 'apache:main' into master
subkanthi Dec 2, 2021
c3030fb
Merge branch 'apache:main' into master
subkanthi Dec 4, 2021
1c3c029
Merge branch 'apache:main' into master
subkanthi Dec 6, 2021
0eb6598
Merge branch 'apache:main' into master
subkanthi Dec 10, 2021
35a344e
Merge branch 'apache:main' into master
subkanthi Dec 12, 2021
8d47486
Merge branch 'apache:main' into master
subkanthi Dec 12, 2021
f450679
Merge branch 'apache:main' into master
subkanthi Dec 14, 2021
67c6f5e
Merge branch 'apache:main' into master
subkanthi Dec 14, 2021
a09d628
Merge branch 'apache:main' into master
subkanthi Dec 14, 2021
1568b72
Merge branch 'apache:main' into master
subkanthi Dec 15, 2021
a89daf1
Merge branch 'apache:main' into master
subkanthi Dec 16, 2021
3c24d50
Merge branch 'apache:main' into master
subkanthi Dec 18, 2021
874b0c2
Merge branch 'apache:main' into master
subkanthi Dec 19, 2021
c07aabf
Merge branch 'apache:main' into master
subkanthi Dec 20, 2021
0f385fc
Merge branch 'apache:main' into master
subkanthi Dec 20, 2021
8bab66c
Merge branch 'apache:main' into master
subkanthi Dec 21, 2021
9f34837
Merge branch 'apache:main' into master
subkanthi Dec 24, 2021
a1dea59
Merge branch 'apache:main' into master
subkanthi Dec 29, 2021
2cbccda
Merge branch 'apache:main' into master
subkanthi Dec 30, 2021
00fbfcb
Merge branch 'apache:main' into master
subkanthi Dec 30, 2021
0c7cc12
Merge branch 'apache:main' into master
subkanthi Dec 31, 2021
3a308dc
Merge branch 'apache:main' into master
subkanthi Dec 31, 2021
c2aa679
Merge branch 'apache:main' into master
subkanthi Jan 4, 2022
a50c41a
Merge branch 'apache:main' into master
subkanthi Jan 5, 2022
2631ca1
Merge branch 'apache:main' into master
subkanthi Jan 6, 2022
8ce7c49
Merge branch 'apache:main' into master
subkanthi Jan 7, 2022
8fb9929
Merge branch 'apache:main' into master
subkanthi Jan 7, 2022
0a9a843
Merge branch 'apache:main' into master
subkanthi Jan 9, 2022
b2a30ba
Merge branch 'apache:main' into master
subkanthi Jan 12, 2022
3e1b58c
Merge branch 'apache:main' into master
subkanthi Jan 15, 2022
41a420c
Merge branch 'apache:main' into master
subkanthi Jan 18, 2022
68c3443
Added logic for kubernetes decorator
subkanthi Jan 18, 2022
d94c4ce
Merge branch 'apache:main' into master
subkanthi Jan 18, 2022
a24a664
Merged from master
subkanthi Jan 18, 2022
402ccaf
Merged from master
subkanthi Jan 18, 2022
01f9319
Merge branch 'apache:main' into master
subkanthi Jan 18, 2022
8faffba
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Jan 19, 2022
e778325
Changes related to decorator composition
subkanthi Jan 19, 2022
9ef371f
Added kubernetes executor
subkanthi Jan 20, 2022
cf2376f
Merge branch 'apache:main' into master
subkanthi Jan 22, 2022
8880c75
Merge branch 'apache:main' into master
subkanthi Jan 23, 2022
857700f
Added jinja template for kubernetes decorator, fixed logic of passing…
subkanthi Jan 23, 2022
1e2ae11
Refactoring of Kubernetes decorator
subkanthi Jan 23, 2022
ee61fa9
Merge branch 'apache:main' into master
subkanthi Jan 24, 2022
8822af9
Merge branch 'apache:main' into master
subkanthi Jan 24, 2022
22a314e
Merge branch 'apache:main' into master
subkanthi Jan 24, 2022
33d6fff
Clean up and added test case
subkanthi Jan 24, 2022
7fd13d7
Clean up and added test case
subkanthi Jan 24, 2022
52adfdd
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Jan 24, 2022
2f54ba1
Merge branch 'apache:main' into master
subkanthi Jan 24, 2022
f1c89a1
Added test case for kubernetes decorator
subkanthi Jan 24, 2022
f63986d
Rebase from master
subkanthi Jan 24, 2022
05162fd
Merge branch 'apache:main' into master
subkanthi Jan 25, 2022
b335457
Merge branch 'apache:main' into master
subkanthi Jan 27, 2022
7d72855
Merge branch 'apache:main' into master
subkanthi Jan 27, 2022
353012f
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Jan 28, 2022
084c712
Removed unused code
subkanthi Jan 28, 2022
031dc2a
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Jan 29, 2022
cb114c3
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Feb 1, 2022
cd5911c
Fix flake8, lint errors
subkanthi Feb 1, 2022
e5f32b8
Changes to move kubernetes decorator logic out of airflow/decorators
subkanthi Feb 1, 2022
eb8ccf1
Reverted back change in __init__.pyi
subkanthi Feb 6, 2022
875be8a
Added kubernetespodopertor parameter to kubernetes decorator
subkanthi Feb 7, 2022
51845b7
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Feb 7, 2022
0fce88e
Merge branch 'master' of github.com:subkanthi/airflow into kubernetes…
subkanthi Feb 10, 2022
aaf1864
Merge branch 'main' into kubernetes_decorator
subkanthi Feb 13, 2022
5e6419b
Merge branch 'main' into kubernetes_decorator
subkanthi Feb 16, 2022
d97ef0c
Removed type definitions in kubernetes decorator
subkanthi Feb 16, 2022
f470507
Merge branch 'main' into kubernetes_decorator
subkanthi Feb 24, 2022
8dd8f4d
Merge branch 'main' into kubernetes_decorator
subkanthi Feb 27, 2022
442e4ce
Cherry pick changes from 21881
subkanthi Mar 1, 2022
49abdd1
Merge branch 'main' into kubernetes_decorator
subkanthi Mar 1, 2022
cf6653b
Update airflow/decorators/__init__.pyi
subkanthi Mar 20, 2022
a1f7ad1
Update airflow/decorators/__init__.pyi
subkanthi Mar 20, 2022
775742a
Update airflow/providers/cncf/kubernetes/decorators/kubernetes.py
subkanthi Mar 20, 2022
83dd407
Update airflow/example_dags/example_kubernetes_decorator.py
subkanthi Mar 20, 2022
cff5197
Merge branch 'main' into kubernetes_decorator
subkanthi Mar 20, 2022
3ca339c
Make @task.kubernetes arguments keyword-only
uranusjr Mar 21, 2022
b4a61da
Merge branch 'main' into kubernetes_decorator
subkanthi Apr 10, 2022
65996b7
Throw exception if image is not passed to kubernetes decorator.
subkanthi Apr 10, 2022
6626060
Add image parameter to kubernetes decorator.
subkanthi Apr 11, 2022
453e32a
Kubernetes decorator -set image parameter as required
subkanthi Apr 11, 2022
ee725a9
Merge branch 'main' into kubernetes_decorator
subkanthi Apr 11, 2022
3b12142
Update airflow/decorators/__init__.pyi
subkanthi Apr 12, 2022
a0d122f
Merge branch 'main' into kubernetes_decorator
subkanthi Apr 13, 2022
821e449
Merge branch 'main' into kubernetes_decorator
subkanthi Apr 27, 2022
890f829
Merge branch 'main' into kubernetes_decorator
subkanthi Jul 5, 2022
f8d54b0
Merge branch 'main' into kubernetes_decorator
subkanthi Jul 5, 2022
47b9062
Merge branch 'main' into kubernetes_decorator
subkanthi Jul 12, 2022
f6edafa
Merge branch 'main' into kubernetes_decorator
subkanthi Jul 12, 2022
00d4f1c
Merge branch 'main' into kubernetes_decorator
subkanthi Jul 12, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions airflow/decorators/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,7 @@ class TaskDecoratorCollection:
Dict will unroll to XCom values with keys as XCom keys. Defaults to False.
:param use_dill: Whether to use dill or pickle for serialization
:param python_command: Python command for executing functions, Default: python3
:param image: Docker image from which to create the container.
If image tag is omitted, "latest" will be used.
:param image: Docker image from which to create the container.(Required)
subkanthi marked this conversation as resolved.
Show resolved Hide resolved
:param api_version: Remote API version. Set to ``auto`` to automatically
detect the server's version.
:param container_name: Name of the container. Optional (templated)
Expand Down Expand Up @@ -248,10 +247,9 @@ class TaskDecoratorCollection:
def kubernetes(
self,
*,
python_callable: Optional[Callable] = None,
multiple_outputs: Optional[bool] = None,
namespace: Optional[str] = None,
image: Optional[str] = None,
image: str,
name: Optional[str] = None,
random_name_suffix: bool = True,
cmds: Optional[List[str]] = None,
Expand Down Expand Up @@ -298,7 +296,6 @@ class TaskDecoratorCollection:

Also accepts any argument that KubernetesPodOperator will via ``kwargs``.

:param python_callable: Function to decorate
:param multiple_outputs: if set, function return value will be
unrolled to multiple XCom values. Dict will unroll to XCom values with keys as XCom keys.
Defaults to False.
Expand Down
6 changes: 1 addition & 5 deletions airflow/providers/cncf/kubernetes/decorators/kubernetes.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@

from kubernetes.client import models as k8s

from airflow import AirflowException
from airflow.decorators.base import DecoratedOperator, TaskDecorator, task_decorator_factory
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.utils.python_virtualenv import remove_task_decorator, write_python_script
Expand Down Expand Up @@ -74,10 +73,7 @@ def __init__(
) -> None:
self.pickling_library = pickle

# Image, name and namespace are all required.
if 'image' not in kwargs:
raise AirflowException("Image required for kubernetes decorator")

# Set defaults for name and namespace.
if 'name' not in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'

Expand Down