Skip to content

Commit

Permalink
[SMALLFIX] Use field property for 'AbstractCmdRunner'
Browse files Browse the repository at this point in the history
### What changes are proposed in this pull request?

Use field property 'mActiveJobs' for 'AbstractCmdRunner' instead of the default value DEFAULT_ACTIVE_JOBS which is hardcoded with 3000.

### Why are the changes needed?

Currently, the number of active jobs are limited at 3000 which is not enough when we have lots of loading tasks on huge cluster. We will add command line args to allow user pass the limit of active jobs (set the value of 'mActiveJobs') the  in the future.

### Does this PR introduce any user facing changes?

N/A
			pr-link: Alluxio#17755
			change-id: cid-cc87ac5731fac22724ca2d84ca3574f2a09033bb
  • Loading branch information
iAlaska authored and codings-dan committed Dec 21, 2023
1 parent daa3b50 commit b72c02f
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ private void submitDistLoad(List<URIStatus> pool, int replication,
Set<String> workerSet, Set<String> excludedWorkerSet,
Set<String> localityIds, Set<String> excludedLocalityIds,
boolean directCache, CmdInfo cmdInfo) {
if (mSubmitted.size() >= DEFAULT_ACTIVE_JOBS) {
if (mSubmitted.size() >= mActiveJobs) {
waitForCmdJob();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ private void copy(AlluxioURI srcPath, AlluxioURI dstPath, boolean overwrite, int
// Submit a child job within a distributed command job.
private void submitDistCp(List<Pair<String, String>> pool, boolean overwrite,
WriteType writeType, CmdInfo cmdInfo) {
if (mSubmitted.size() >= DEFAULT_ACTIVE_JOBS) {
if (mSubmitted.size() >= mActiveJobs) {
waitForCmdJob();
}

Expand Down

0 comments on commit b72c02f

Please sign in to comment.