Skip to content

Commit

Permalink
Fix local cache identifier to solve the local cache consistent issue
Browse files Browse the repository at this point in the history
### What changes are proposed in this pull request?

Fix local cache identifier

### Why are the changes needed?

If not including `modification time`, it may have consistency issues.

### Does this PR introduce any user facing changes?

NO

			pr-link: #17514
			change-id: cid-c9c2e91e53d4a95d4f3a165918fc23c02f623891
  • Loading branch information
secfree authored Jul 3, 2023
1 parent 1e5eebe commit cfcda18
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import alluxio.client.file.cache.LocalCacheFileInStream;
import alluxio.client.file.cache.filter.CacheFilter;
import alluxio.conf.AlluxioConfiguration;
import alluxio.conf.PropertyKey;
import alluxio.metrics.MetricsConfig;
import alluxio.metrics.MetricsSystem;
import alluxio.wire.FileInfo;
Expand Down Expand Up @@ -130,8 +131,15 @@ public FSDataInputStream open(Path path, int bufferSize) throws IOException {
.setGroup(externalFileStatus.getGroup());
// FilePath is a unique identifier for a file, however it can be a long string
// hence using md5 hash of the file path as the identifier in the cache.
CacheContext context = CacheContext.defaults().setCacheIdentifier(
md5().hashString(externalFileStatus.getPath().toString(), UTF_8).toString());
String cacheIdentifier;
if (mAlluxioConf.getBoolean(PropertyKey.USER_CLIENT_CACHE_IDENTIFIER_INCLUDE_MTIME)) {
// include mtime to avoid consistency issues if the file may update
cacheIdentifier = md5().hashString(externalFileStatus.getPath().toString()
+ externalFileStatus.getModificationTime(), UTF_8).toString();
} else {
cacheIdentifier = md5().hashString(externalFileStatus.getPath().toString(), UTF_8).toString();
}
CacheContext context = CacheContext.defaults().setCacheIdentifier(cacheIdentifier);
URIStatus status = new URIStatus(info, context);
return open(status, bufferSize);
}
Expand Down
10 changes: 10 additions & 0 deletions core/common/src/main/java/alluxio/conf/PropertyKey.java
Original file line number Diff line number Diff line change
Expand Up @@ -6229,6 +6229,14 @@ public String toString() {
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.CLIENT)
.build();
public static final PropertyKey USER_CLIENT_CACHE_IDENTIFIER_INCLUDE_MTIME =
booleanBuilder(Name.USER_CLIENT_CACHE_IDENTIFIER_INCLUDE_MTIME)
.setDefaultValue(false)
.setDescription("If this is enabled, client-side cache will include modification time "
+ "while calculating the identifier of a file.")
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.CLIENT)
.build();

public static final PropertyKey USER_CLIENT_REPORT_VERSION_ENABLED =
booleanBuilder(Name.USER_CLIENT_REPORT_VERSION_ENABLED)
Expand Down Expand Up @@ -8879,6 +8887,8 @@ public static final class Name {
"alluxio.user.client.cache.timeout.duration";
public static final String USER_CLIENT_CACHE_TIMEOUT_THREADS =
"alluxio.user.client.cache.timeout.threads";
public static final String USER_CLIENT_CACHE_IDENTIFIER_INCLUDE_MTIME =
"alluxio.user.client.cache.include.mtime";
public static final String USER_CLIENT_REPORT_VERSION_ENABLED =
"alluxio.user.client.report.version.enabled";
public static final String USER_CONF_CLUSTER_DEFAULT_ENABLED =
Expand Down

0 comments on commit cfcda18

Please sign in to comment.