Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename container_image to image for improved UX #3094

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

JiangJiaWei1103
Copy link
Contributor

@JiangJiaWei1103 JiangJiaWei1103 commented Jan 25, 2025

Tracking issue

Closes flyteorg/flyte#6140.

Why are the changes needed?

To enhance the user experience, the concept of container should be abstracted from flytekit users.

What changes were proposed in this pull request?

At this early stage, we propose to focus on the user-facing codebase and support both image and container_image for backward compatibility. There exist three cases:

  1. Both image and container_image are specified: Raise an error
  2. container_image is used: Warn users about the future deprecation
  3. image is used: Use image directly

In the future, we can revisit whether modifying the developer-facing code is beneficial when it becomes more relevant.

How was this patch tested?

This patch has been tested using modified and newly added unit tests.

Setup process

git clone https://github.com/flyteorg/flytekit.git
gh pr checkout 3094
make setup && pip install -e .

Run all unit tests using the following command:

make unit_test

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

NA

Docs link

Please refer to flyteorg/flyte#6211

Summary by Bito

This PR implements both the renaming of 'container_image' parameter to 'image' across Flytekit codebase and introduces comprehensive error handling improvements. Changes include deprecation warnings, secure handling of pip arguments, improved async operations, and better Kubernetes resource validation. The implementation maintains backward compatibility while adding centralized warning logic and error handling for better runtime stability.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

1. Support both `image` and `container_image` for backward compatibility
2. Modify the core decorator used for any task type
    * We focus on the user-facing inteface

Signed-off-by: JiangJiaWei1103 <[email protected]>
@flyte-bot
Copy link
Contributor

Code Review Agent Run Status

  • Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].

Copy link

codecov bot commented Jan 29, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.58%. Comparing base (22acc2c) to head (896553e).
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3094      +/-   ##
==========================================
+ Coverage   80.01%   83.58%   +3.57%     
==========================================
  Files         318        3     -315     
  Lines       27075      195   -26880     
  Branches     2779        0    -2779     
==========================================
- Hits        21663      163   -21500     
+ Misses       4647       32    -4615     
+ Partials      765        0     -765     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@flyte-bot
Copy link
Contributor

Code Review Agent Run Status

  • Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].

@flyte-bot
Copy link
Contributor

Code Review Agent Run Status

  • Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].

@JiangJiaWei1103 JiangJiaWei1103 changed the title [WIP] Rename container_image to image for improved UX Rename container_image to image for improved UX Feb 2, 2025
@JiangJiaWei1103 JiangJiaWei1103 marked this pull request as ready for review February 2, 2025 16:22
@@ -584,6 +606,24 @@ async def eager_workflow(x: int) -> int:
async def eager_workflow(x: int) -> int:
...
"""
# Rename the `container_image` parameter to `image` for improved user experience.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should image be added to PythonAutoContainerTask and we do the validation there? This way @eager (through EagerAsyncPythonFunctionTask) and @task will get the check.

For backward compatible, I would have PythonAutoContainerTask._image, but have ._container_image and container_image be a mirror of it:

class PythonAutoContainerTask:
    def __init__(self, container_image, image):
        # validate image and container_image
        self._image = container_image
        
    @property
    def container_image(self):
        return self._image

    @property
    def _container_image(self):
        return self._image
        
    @_container_image.setter
    def _container_image(self, value):
        self._image = value

    @property
    def image(self):
        return self._image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both work fine for me. Thanks for your suggestion--I learned a lot!

I have two questions:

  1. Is it common practice to make a class’s internal attribute (like _container_image here) accessible and modifiable from outside the class?
    • I assume we do this to avoid breaking users’ code.
  2. Would it be good to add a setter for image?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume we do this to avoid breaking users’ code.

I'm mostly trying to keep backward compatibility. I see bunch of code paths that does obj._container_image = . (I do not really like this practice, but that's the current state of the code base)

Would it be good to add a setter for image?

I'll say it's not worth it here. If we keep the obj._container_image = practice, then future users can do obj._image = .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! Everything is now perfectly clear.

@flyte-bot
Copy link
Contributor

flyte-bot commented Feb 2, 2025

Code Review Agent Run #e396c5

Actionable Suggestions - 3
  • flytekit/core/type_engine.py - 1
    • Consider maintaining parameter name consistency · Line 383-383
  • flytekit/core/task.py - 2
    • Consider consolidating duplicate image parameters · Line 108-108
    • Consider consolidating duplicate image parameters · Line 148-148
Review Details
  • Files reviewed - 9 · Commit Range: ff7b2c5..f2ddb80
    • flytekit/core/task.py
    • flytekit/core/type_engine.py
    • tests/flytekit/integration/jupyter/test_notebook_run.py
    • tests/flytekit/unit/cli/pyflyte/image_spec_wf.py
    • tests/flytekit/unit/core/test_array_node_map_task.py
    • tests/flytekit/unit/core/test_python_function_task.py
    • tests/flytekit/unit/core/test_serialization.py
    • tests/flytekit/unit/core/test_task.py
    • tests/flytekit/unit/remote/test_remote.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

@flyte-bot
Copy link
Contributor

flyte-bot commented Feb 2, 2025

Changelist by Bito

This pull request implements the following key changes.

Key Change Files Impacted
Feature Improvement - Rename container_image parameter to image for better UX

python_auto_container.py - Implemented image parameter with deprecation warning for container_image

task.py - Updated task decorator to use image parameter instead of container_image

type_engine.py - Updated custom image reference to use new image parameter

Testing - Add comprehensive tests for image parameter changes

test_task.py - Added new test cases for image parameter usage and deprecation warnings

test_serialization.py - Updated serialization tests to use new image parameter

test_notebook_run.py - Updated notebook tests to use new image parameter

image_spec_wf.py - Updated image spec workflow tests

test_remote.py - Updated remote execution tests with new image parameter

@@ -380,7 +380,7 @@ class DC:
a: Union[int, bool, str, float]
b: Union[int, bool, str, float]
@task(container_image=custom_image)
@task(image=custom_image)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider maintaining parameter name consistency

Consider updating the parameter name from image to container_image to maintain backward compatibility and avoid potential breaking changes for existing code.

Code suggestion
Check the AI-generated fix before applying
Suggested change
@task(image=custom_image)
@task(container_image=custom_image)

Code Review Run #e396c5


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

@@ -104,6 +105,7 @@ def task(
interruptible: Optional[bool] = ...,
deprecated: str = ...,
timeout: Union[datetime.timedelta, int] = ...,
image: Optional[Union[str, ImageSpec]] = ...,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider consolidating duplicate image parameters

Consider consolidating the image and container_image parameters since they appear to serve the same purpose. Having two parameters for the same functionality could lead to confusion.

Code suggestion
Check the AI-generated fix before applying
 -    image: Optional[Union[str, ImageSpec]] = ...,
 -    container_image: Optional[Union[str, ImageSpec]] = ...,
 +    container_image: Optional[Union[str, ImageSpec]] = ...,  # Use this instead of image parameter
 +    image: Optional[Union[str, ImageSpec]] = None,  # Deprecated: Use container_image instead

Code Review Run #e396c5


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

@@ -143,6 +145,7 @@ def task(
interruptible: Optional[bool] = ...,
deprecated: str = ...,
timeout: Union[datetime.timedelta, int] = ...,
image: Optional[Union[str, ImageSpec]] = ...,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider consolidating duplicate image parameters

Consider consolidating the image parameter with the existing container_image parameter to avoid semantic duplication, as both appear to serve the same purpose of specifying container images.

Code suggestion
Check the AI-generated fix before applying
 -    image: Optional[Union[str, ImageSpec]] = ...,
 -    container_image: Optional[Union[str, ImageSpec]] = ...,
 +    container_image: Optional[Union[str, ImageSpec]] = ...,  # Also accepts image parameter for backward compatibility

Code Review Run #e396c5


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

1. Validate `image` and `container_image` in `PythonAutoContainerTask`'s
    initializer
2. Maintain backward compatibility within `PythonAutoContainerTask`

Signed-off-by: JiangJiaWei1103 <[email protected]>
@flyte-bot
Copy link
Contributor

flyte-bot commented Feb 3, 2025

Code Review Agent Run #5e8094

Actionable Suggestions - 2
  • flytekit/core/task.py - 1
  • flytekit/core/python_auto_container.py - 1
    • Consider consolidating duplicate image properties · Line 161-171
Review Details
  • Files reviewed - 2 · Commit Range: f2ddb80..bb6e187
    • flytekit/core/python_auto_container.py
    • flytekit/core/task.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Comment on lines +381 to 382
image=image,
container_image=container_image,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider consolidating image parameters

Consider consolidating the image parameters. Currently both image and container_image are being passed, which could lead to confusion about precedence. Since container_image is deprecated, it would be clearer to use only image.

Code suggestion
Check the AI-generated fix before applying
Suggested change
image=image,
container_image=container_image,
image=image,

Code Review Run #5e8094


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

Comment on lines 161 to 171
"""Deprecated, please use `image` instead."""
return self._image

@property
def _container_image(self) -> Optional[Union[str, ImageSpec]]:
"""Deprecated, please use `image` instead."""
return self._image

@_container_image.setter
def _container_image(self, image: Optional[Union[str, ImageSpec]]):
self._image = image
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider consolidating duplicate image properties

Consider consolidating the duplicate property definitions for container_image and _container_image. Both properties return self._image and have the same docstring.

Code suggestion
Check the AI-generated fix before applying
 -    def container_image(self) -> Optional[Union[str, ImageSpec]]:
 -        """Deprecated, please use `image` instead."""
 -        return self._image
 -
 -    @property
 -    def _container_image(self) -> Optional[Union[str, ImageSpec]]:
 -        """Deprecated, please use `image` instead."""
 -        return self._image
 -
 -    @_container_image.setter
 -    def _container_image(self, image: Optional[Union[str, ImageSpec]]):
 +    @property
 +    def _container_image(self) -> Optional[Union[str, ImageSpec]]:
 +        """Deprecated, please use `image` instead."""
 +        return self._image

Code Review Run #5e8094


Is this a valid issue, or was it incorrectly flagged by the Agent?

  • it was incorrectly flagged

pingsutw
pingsutw previously approved these changes Feb 3, 2025
thomasjpfan
thomasjpfan previously approved these changes Feb 4, 2025
Copy link
Member

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit on adding a comment

return self._image

@_container_image.setter
def _container_image(self, image: Optional[Union[str, ImageSpec]]):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _container_image(self, image: Optional[Union[str, ImageSpec]]):
def _container_image(self, image: Optional[Union[str, ImageSpec]]):
"""Deprecated, please use `image` instead."""
# This setter is for backward compatibility, so that setting `_container_image`
# will adjust the new `_image` parameter directly.

Copy link
Contributor Author

@JiangJiaWei1103 JiangJiaWei1103 Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Thomas,

I’ve just added it. Thanks for your guidance!

@JiangJiaWei1103 JiangJiaWei1103 dismissed stale reviews from thomasjpfan and pingsutw via e66743f February 4, 2025 15:31
@flyte-bot
Copy link
Contributor

flyte-bot commented Feb 4, 2025

Code Review Agent Run #acb303

Actionable Suggestions - 0
Review Details
  • Files reviewed - 1 · Commit Range: bb6e187..e66743f
    • flytekit/core/python_auto_container.py
  • Files skipped - 0
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

@flyte-bot
Copy link
Contributor

flyte-bot commented Feb 6, 2025

Code Review Agent Run #74c115

Actionable Suggestions - 0
Additional Suggestions - 10
  • flytekit/models/security.py - 2
    • Consider adding env_var validation check · Line 45-45
    • Consider validating env_var for empty string · Line 73-73
  • tests/flytekit/integration/remote/workflows/basic/get_secret.py - 1
    • Consider explicit error for missing env var · Line 19-20
  • flytekit/deck/deck.py - 1
    • Add error handling for deck publishing · Line 96-96
  • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_sensor.py - 1
    • Add context manager for async task · Line 25-29
  • plugins/flytekit-k8sdataservice/setup.py - 1
    • Consider more specific kubernetes version range · Line 7-7
  • flytekit/image_spec/image_spec.py - 1
    • Consider adding pip args validation · Line 86-86
  • tests/flytekit/unit/core/test_data_persistence.py - 1
    • Consider strengthening test assertion specificity · Line 237-238
  • tests/flytekit/unit/types/structured_dataset/test_structured_dataset.py - 1
  • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/manager.py - 1
    • Consider breaking down container creation method · Line 66-82
Review Details
  • Files reviewed - 74 · Commit Range: e66743f..896553e
    • .pre-commit-config.yaml
    • Dockerfile.agent
    • docs/source/plugins/k8sstatefuldataservice.rst
    • flytekit/bin/entrypoint.py
    • flytekit/clients/friendly.py
    • flytekit/clients/raw.py
    • flytekit/clis/sdk_in_container/serve.py
    • flytekit/core/base_task.py
    • flytekit/core/context_manager.py
    • flytekit/core/data_persistence.py
    • flytekit/core/node.py
    • flytekit/core/python_function_task.py
    • flytekit/core/resources.py
    • flytekit/core/type_engine.py
    • flytekit/deck/deck.py
    • flytekit/image_spec/default_builder.py
    • flytekit/image_spec/image_spec.py
    • flytekit/interaction/parse_stdin.py
    • flytekit/loggers.py
    • flytekit/models/core/workflow.py
    • flytekit/models/domain.py
    • flytekit/models/security.py
    • flytekit/models/task.py
    • flytekit/remote/remote.py
    • flytekit/remote/remote_fs.py
    • flytekit/tools/fast_registration.py
    • flytekit/tools/translator.py
    • flytekit/types/directory/types.py
    • flytekit/types/structured/structured_dataset.py
    • plugins/flytekit-envd/flytekitplugins/envd/image_builder.py
    • plugins/flytekit-envd/tests/test_image_spec.py
    • plugins/flytekit-k8sdataservice/dev-requirements.txt
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/__init__.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/agent.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/kube_config.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/k8s/manager.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/sensor.py
    • plugins/flytekit-k8sdataservice/flytekitplugins/k8sdataservice/task.py
    • plugins/flytekit-k8sdataservice/setup.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/k8s/test_kube_config.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/k8s/test_manager.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_agent.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_sensor.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/test_task.py
    • plugins/flytekit-k8sdataservice/tests/k8sdataservice/utils/test_resources.py
    • plugins/flytekit-k8sdataservice/utils/infra.py
    • plugins/flytekit-k8sdataservice/utils/resources.py
    • plugins/flytekit-omegaconf/flytekitplugins/omegaconf/dictconfig_transformer.py
    • plugins/flytekit-omegaconf/tests/test_dictconfig_transformer.py
    • plugins/flytekit-ray/flytekitplugins/ray/task.py
    • plugins/flytekit-ray/setup.py
    • plugins/flytekit-ray/tests/test_ray.py
    • plugins/setup.py
    • pydoclint-errors-baseline.txt
    • pyproject.toml
    • tests/flytekit/clis/sdk_in_container/test_serve.py
    • tests/flytekit/integration/remote/test_remote.py
    • tests/flytekit/integration/remote/workflows/basic/get_secret.py
    • tests/flytekit/integration/remote/workflows/basic/sd_attr.py
    • tests/flytekit/unit/core/image_spec/test_default_builder.py
    • tests/flytekit/unit/core/test_array_node_map_task.py
    • tests/flytekit/unit/core/test_data_persistence.py
    • tests/flytekit/unit/core/test_dataclass.py
    • tests/flytekit/unit/core/test_generice_idl_type_engine.py
    • tests/flytekit/unit/core/test_list.py
    • tests/flytekit/unit/core/test_map_task.py
    • tests/flytekit/unit/core/test_node_creation.py
    • tests/flytekit/unit/core/test_resources.py
    • tests/flytekit/unit/core/test_type_engine.py
    • tests/flytekit/unit/core/test_unions.py
    • tests/flytekit/unit/deck/test_deck.py
    • tests/flytekit/unit/extras/pydantic_transformer/test_pydantic_basemodel_transformer.py
    • tests/flytekit/unit/test_translator.py
    • tests/flytekit/unit/types/structured_dataset/test_structured_dataset.py
  • Files skipped - 2
    • .github/workflows/pythonbuild.yml - Reason: Filter setting
    • plugins/flytekit-k8sdataservice/README.md - Reason: Filter setting
  • Tools
    • Whispers (Secret Scanner) - ✔︎ Successful
    • Detect-secrets (Secret Scanner) - ✔︎ Successful
    • MyPy (Static Code Analysis) - ✔︎ Successful
    • Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by Bito Logo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In review
Development

Successfully merging this pull request may close these issues.

[flytekit] Polish - Container ux
4 participants