Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/support large dml split #384

Merged
merged 54 commits into from
Jan 19, 2024
Merged

Conversation

newborn22
Copy link
Collaborator

@newborn22 newborn22 commented Dec 12, 2023

Description

Related Issue(s)

#403

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

@newborn22 newborn22 force-pushed the feature/support_large_DML_split branch from 5085f1c to c547adf Compare December 12, 2023 09:17
@earayu earayu marked this pull request as draft December 19, 2023 05:00
@newborn22 newborn22 force-pushed the feature/support_large_DML_split branch 2 times, most recently from a69f4dc to c8dfdc5 Compare December 27, 2023 02:23
@newborn22 newborn22 marked this pull request as ready for review January 2, 2024 03:22
@newborn22 newborn22 requested review from gerayking and earayu January 2, 2024 03:24
@newborn22 newborn22 force-pushed the feature/support_large_DML_split branch from fe0fc26 to 111fb05 Compare January 2, 2024 11:18
@earayu
Copy link
Collaborator

earayu commented Jan 2, 2024

Need more Unit Tests.
For adding more Unit Tests, you may need to refactor some functions.

@newborn22 newborn22 force-pushed the feature/support_large_DML_split branch 2 times, most recently from 741e189 to 83e7beb Compare January 10, 2024 05:23
@@ -193,27 +196,29 @@ func NewTabletServer(name string, config *tabletenv.TabletConfig, topoServer *to
tsv.branchWatch = NewBranchWatcher(tsv, tsv.config.DB.DbaWithDB())

tsv.onlineDDLExecutor = onlineddl.NewExecutor(tsv, alias, topoServer, tsv.lagThrottler, tabletTypeFunc, tsv.onlineDDLExecutorToggleTableBuffer)
tsv.dmlJonController = jobcontroller.NewJobController("non_transactional_dml_jobs", tabletTypeFunc, tsv, tsv.lagThrottler)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace this magic string

}

// todo,feat 可以增加并发Job数的限制
// 调用该函数时外部必须拿tableMutex锁和workingTablesMutex锁
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users should be able to change the 'Job level concurrency'.
Need to add a parameter for it.

}
}

func (jc *JobController) tableGC(ctx context.Context, uuid, tableSchema, batchInfoTable, statusSetTime string) error {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should refer tableGC module.

Signed-off-by: newborn22 <[email protected]>
…sed on single table, single int pk

Signed-off-by: newborn22 <[email protected]>
Signed-off-by: newborn22 <[email protected]>
newborn22 and others added 22 commits January 18, 2024 13:15
Signed-off-by: newborn22 <[email protected]>
…bit sql without where clause and with limit or order by clause;

Signed-off-by: newborn22 <[email protected]>
…uery after calling setQuery; replace sleep with ticker

Signed-off-by: newborn22 <[email protected]>
…ast node instead of string operations

Signed-off-by: newborn22 <[email protected]>
…eclare unsupported PK type; fix failpolicy bugs

Signed-off-by: newborn22 <[email protected]>
Signed-off-by: newborn22 <[email protected]>
Signed-off-by: newborn22 <[email protected]>
@newborn22 newborn22 force-pushed the feature/support_large_DML_split branch from a5e3263 to 6dc0252 Compare January 18, 2024 05:17
Copy link
Collaborator

@earayu earayu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@earayu earayu merged commit 8b33761 into main Jan 19, 2024
17 checks passed
@earayu earayu deleted the feature/support_large_DML_split branch January 19, 2024 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants