Support for serving disk persisted RDDs to enable efficient Spark DRA

[pinireisman]

When using in Spark the ExternalShuffleService - it is able to serve RDDs that have been persisted to disk from the shuffle service (as is described in SPARK-27677 ) thus allowing the executor that was responsible for that persisted data (rdd partitions) to be removed by the Dynamic Resource Allocation mechanism without losing access to the persisted RDD partitions.

Can Celeborn be used to do the same thing? It would be very beneficial if these RDD partitions can be served by Celeborn similarly as is done by the External Shuffle Service.

Does such a feature exist in Celeborn? Is anyone working on such a features?