[spark] Adjust ExpireSnapshotsProcedure param type #4059
Conversation
|
help cc @JingsongLi |
| ProcedureParameter.optional("max_deletes", IntegerType) | ||
| ProcedureParameter.optional("older_than", StringType), | ||
| ProcedureParameter.optional("max_deletes", IntegerType), | ||
| ProcedureParameter.optional("time_retained", StringType) |
There was a problem hiding this comment.
older_than and time_retained? Looks not easy to understand, we should just keep one.
There was a problem hiding this comment.
Because there is currently compatibility issue. I recommend using a script to address the issue of periodic calls when invoking from the outer layer.
There was a problem hiding this comment.
older_than and time_retained? Looks not easy to understand, we should just keep one.
These two parameters can not be used together.
There was a problem hiding this comment.
Because there is currently compatibility issue. I recommend using a script to address the issue of periodic calls when invoking from the outer layer.
When using a script to calls periodically, we need to adjust older_than parameter each time. I think it is necessary to keep time_retained, what do you think?
There was a problem hiding this comment.
Thanks. @JingsongLi Removed time_retained
askwang
force-pushed
the
expire-snapshot-param
branch
from
df6c4a4
to
6400f12
Compare
| Duration.ofMillis(System.currentTimeMillis() - olderThanMills)); | ||
| if (!StringUtils.isNullOrWhitespaceOnly(olderThanStr)) { | ||
| long olderThanMills; | ||
| // forward compatibility for timestamp type |
There was a problem hiding this comment.
How to forward compatibility for timestamp type? It is a string instead of a timestamp.
There was a problem hiding this comment.
In default, the value of timestamp type is like "1724664000000", it's actually a string that contains numerics. And get it by
Long olderThanMills = args.isNullAt(3) ? null : args.getLong(3) / 1000;
There was a problem hiding this comment.
change param older_than type from TimeStampType to StringType, TimeStampType is not easy to understand and use.
Is this related? Can you show the use cases? How to improve the usage.
There was a problem hiding this comment.
got, has only retain StringType
Purpose
older_thantype from TimeStampType to StringType.Tests
API and Format
Documentation