Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved debug on migrations #181

Closed
pdbossman opened this issue Jul 19, 2024 · 2 comments · Fixed by #199
Closed

Improved debug on migrations #181

pdbossman opened this issue Jul 19, 2024 · 2 comments · Fixed by #199
Assignees
Labels
enhancement New feature or request

Comments

@pdbossman
Copy link
Contributor

There are different mandatory and optional input parameters to a scylla migration.

It would be useful at the beginning of migration to dump out input parameters and actual parameters used.
For example, aws region for DynamoDB, scanSegments.

Right now, it is guess work what these parameters are and if they were visible, the diagnosis of problems would be much faster.

@julienrf julienrf added the enhancement New feature or request label Jul 19, 2024
@julienrf julienrf self-assigned this Jul 23, 2024
@julienrf
Copy link
Collaborator

julienrf commented Aug 14, 2024

I reviewed the way we use the configuration parameters and discovered that some of them are ineffective (e.g., setting scanSegments in the target does nothing, it works only if set to the source table).

Other than that, there are already logs that describe which parameters are used to configure the Hadoop job properties. See e.g. these logs or these logs.

@pdbossman would you consider this issue fixed with the following plan?

  • Remove the unused configuration parameters, which currently contribute to your confusion,
  • Add more logs about characteristics of the migration that come from the config vs characteristics that are computed with default values (e.g here the supplied AWS region is used)

Or, is there anything else you need to improve debugging on migrations?

@julienrf
Copy link
Collaborator

See PR #199.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants