From 5e1504487d9c5c0d210bc231669f68049c0ca213 Mon Sep 17 00:00:00 2001 From: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Date: Wed, 24 Jul 2024 11:32:17 +0100 Subject: [PATCH] Add documentation for how plugin CLI commands can be made lazy loaded (#4027) * Add docs for making plugins lazy loaded Signed-off-by: Ankita Katiyar * Suggestions from reviews: Signed-off-by: Ankita Katiyar * Small changes Signed-off-by: Ankita Katiyar --------- Signed-off-by: Ankita Katiyar --- docs/source/extend_kedro/plugins.md | 51 +++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/docs/source/extend_kedro/plugins.md b/docs/source/extend_kedro/plugins.md index 11ad0a72b3..0332600e45 100644 --- a/docs/source/extend_kedro/plugins.md +++ b/docs/source/extend_kedro/plugins.md @@ -22,7 +22,7 @@ def commands(): pass -@commands.command() +@commands.command(name="to_json") @click.pass_obj def to_json(metadata): """Display the pipeline in JSON format""" @@ -48,7 +48,6 @@ Commands must be provided as [`click` `Groups`](https://click.palletsprojects.co The `click Group` will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function. - ## Project context When they run, plugins may request information about the current project by creating a session and loading its context: @@ -78,7 +77,53 @@ Plugins may also add commands to the Kedro CLI, which supports two types of comm ## Suggested command convention -We use the following command convention: `kedro `, with `kedro ` acting as a top-level command group. This is our suggested way of structuring your plugin bit it is not necessary for your plugin to work. +We use the following command convention: `kedro `, with `kedro ` acting as a top-level command group. This is our suggested way of structuring your plugin but it is not necessary for your plugin to work. + +## Advanced: Lazy loading of plugin commands + +If you are developing a plugin with a large set of CLI commands or with certain large libraries that are slow to import but are used by a small subset of commands, consider using lazy loading of these commands. This can significantly improve the performance of the plugin as well as the overall performance of Kedro CLI. To do this, you can follow [the instructions on the +`click` documentation on lazy loading of commands](https://click.palletsprojects.com/en/8.1.x/complex/#lazily-loading-subcommands). From Kedro 0.19.7, the [Kedro commands are declared as lazy loaded command groups](https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/cli.py) that you can use as a reference for the implementation. + +Consider the previous example of the `kedrojson` plugin. Suppose the plugin has two commands, `kedro to_json pipelines` and `kedro to_json nodes`. The `to_json pipelines` command is used more frequently than the `to_json nodes` command and the `to_json nodes` command requires a large library to be imported. In this case, you can define your commands to be lazily loaded with delayed imports as follows: + +In `kedrojson/plugin.py`: +```python +import click +from kedro.framework.project import pipelines +from kedro.framework.cli.utils import LazyGroup + +@click.group() +def commands(): + pass + +@commands.group( + name="to_json", + cls=LazyGroup, + lazy_subcommands={ + "nodes": "kedrojson.plugin.nodes", + "pipelines": "kedrojson.plugin.pipelines" + } +) +def to_json(): + """Convert Kedro nodes and pipelines to JSON""" + pass + +@click.command(name="nodes") +def nodes(): + """Convert Kedro nodes to JSON""" + import some_large_library + print("Converting nodes to JSON") + ... + +@click.command("pipelines") +def pipelines(): + """Convert Kedro pipelines to JSON""" + print("Converting pipelines to JSON") + ... +``` + +The loading of the individual `nodes` and `pipelines` commands and subsequently the imports of the large libraries will be delayed until the respective commands are called. + ## Hooks