Update DataCatalog private methods with DataCatalog 2.0 API public methods #2274
Conversation
Signed-off-by: Sajid Alam <[email protected]>
Signed-off-by: Sajid Alam <[email protected]>
SajidAlamQB
requested review from
rashidakanchwala and
ravi-kumar-pilla
as code owners
Signed-off-by: Sajid Alam <[email protected]>
| @@ -6,6 +6,13 @@ | |||
|
|
|||
| from kedro.io import DataCatalog | |||
There was a problem hiding this comment.
does it make sense to import this only as a fallback if there's an importError when we are in an older kedro version?
There was a problem hiding this comment.
I think we still need DataCatalog as the “baseline” import so our code is backward compatible with older Kedro (which doesn’t have KedroDataCatalog)
There was a problem hiding this comment.
should the line 7 be inside except ImportError at line 13 ? Normally that is how we were doing bc before. Wouldn't this raise error if DataCatalog is not available in future ?
Signed-off-by: Sajid Alam <[email protected]>
| dataset = self._catalog.get(dataset_name) | ||
| else: | ||
| dataset = self._catalog._get_dataset(dataset_name) | ||
| metadata = getattr(dataset, "metadata", None) |
There was a problem hiding this comment.
how are we handling metadata in 2.0 ?
| @@ -91,7 +98,10 @@ def resolve_dataset_factory_patterns( | |||
|
|
|||
| for dataset_name in datasets: | |||
| try: | |||
| catalog._get_dataset(dataset_name, suggest=False) | |||
| if IS_DATACATALOG_2 and isinstance(catalog, KedroDataCatalog): | |||
There was a problem hiding this comment.
Not regarding this PR but I remember Elena mentioned resolve_dataset_factory_patterns will happen on kedro side. Do we still need to have this function here? Can we call this function only when it is not IS_DATACATALOG_2 ?
|
Hi @SajidAlamQB , Nice work ! I have left few comments. May be you can also update release note for tracking this work ? |
Description
Related to: #2273
This PR replaces references to the private
_get_dataset()with Kedro’s publiccatalog.get()method.To maintain backward compatibility for older Kedro versions that do not include
KedroDataCatalog, introduced a version/feature check that falls back to_get_dataset()only if necessary.Development notes
IS_DATACATALOG_2flag to detect whetherKedroDataCatalogis available._get_dataset()calls withcatalog.get()._get_dataset()is used only for older Kedro versions.Checklist
RELEASE.mdfile