Lazy loading for IncrementalDataSet

[antoinebon]

Is there a particular reason why IncrementalDataSet is not lazy loaded like PartitionedDataset?

Also the type signature for IncrementalDataSet._load seems incorrect. I would expect a return type similar to Dict[str, object] rather than Dict[str, Callable[[], Any]] :

    def _load(self) -> Dict[str, Callable[[], Any]]:
        partitions = {}

        for partition in self._list_partitions():
            partition_id = self._path_to_partition(partition)
            kwargs = deepcopy(self._dataset_config)
            # join the protocol back since PySpark may rely on it
            kwargs[self._filepath_arg] = self._join_protocol(partition)
            partitions[partition_id] = self._dataset_type(  # type: ignore
                **kwargs
            ).load()

        return partitions

[antonymilne]

This is a great question, and I had to look back through some old git history to find out the answer...
Note PartitionedDataSetWithCheckpoint was the in-development name of IncrementalDataSet.
So it seems that the problem here is the type signature rather than the behaviour itself.