Feature Area

/area backend
/area sdk

What feature would you like to see?

Kubeflow Pipelines has a caching feature that allows users to avoid re-running pipeline components (steps in the pipeline) if the system detects that such a component has previously run and its outputs (artifacts) could be reused. The goal is to save time and computation.

By default, the KFP compiler defaults to setting caching enabled on every Component/Task unless the pipeline author calls

task.set_caching_options(False)

In other words:

@dsl.pipeline(name='iris-training-pipeline')
def my_pipeline():
   task_1 = create_dataset()
   task_2 = create_dataset()
   task_1.set_caching_options(False)   <-- task 1 won’t enable caching, but task 2 will ...
                                           even though the author didn’t specify anything about task 2!

Caching disabled is a much more reasonable default.

DSL Example

Caching is controlled on each individual pipeline Component / Task.
Here is example KFP DSL code that disables caching for a single task:

@dsl.pipeline(name='iris-training-pipeline')
def my_pipeline():
   create_dataset_task = create_dataset()
   create_dataset_task.set_caching_options(False)      <-- this task won’t enable caching

Today, the KFP compiler defaults to setting caching enabled on every Component/Task unless the pipeline author calls task.set_caching_options(False)

In other words:

@dsl.pipeline(name='iris-training-pipeline')
def my_pipeline():
   task_1 = create_dataset()
   task_2 = create_dataset()
   task_1.set_caching_options(False)
   # task 1 won’t try to use the cache, but task 2 will ...
   # even though the author didn’t specify anything about task 2!

When we are done with this feature, this will be true:

@dsl.pipeline(name='iris-training-pipeline')
def my_pipeline():
   task_1 = create_dataset()
   task_2 = create_dataset()
   task_3 = create_dataset()
   task_3.set_caching_options(True)
   # tasks 1 and 2 don’t try to use the cache. Task 3 does try to use the cache.

What is the use case or pain point?

We need to fix the KFP compiler to stop enabling caching by default (by setting task.set_caching_options(True)) if the user didn’t ask for that. As described above, the effect of this behavior is that everything tries to use the cache by default, even though caching is disabled by default in the backend.

This might be a significant change, we wish to have a discussion with the KFP community, get consensus on this update and then proceed with making changes. Find a related issue here: #10839

Love this idea? Give it a 👍.