Stop data-diff when maximum time or # different records is exceeded

[JCZuurmond]

Is your feature request related to a problem? Please describe.

We run data-diff for many tables. Sometimes there are a lot of differences between the diffed tables. If so, the data diff for this tablepair might take a very long time (multiple hours). I prefer to skip this diff at a certain point, e.g., when a maximum diff time or # different records is exceeded. For such a diff, I do not care which records differ precisely, I am ok with knowing that this table is very off.

Describe the solution you'd like

Define a:

• maximum diff time
• OR, a maximum # different records
• OR, a maximum % different records

If this threshold is exceeded, the diff is aborted, with a WARNING or ERROR message, and maybe an Exception.

Describe alternatives you've considered

I run data-diff programmatically and built this feature myself in the Python script that calls data-diff. This did not work as I hoped because data-diff uses a ThreadPool that continued with the diff after I broke out of the diff_tables iterable.

Additional context

[github-actions]

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[JCZuurmond]

I am still interested in this issue

[dlawin]

Agreed this would be very useful.

We just added some automation here yesterday to help us wrangle the open issues list, thanks for bearing with us

[github-actions]

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

[JCZuurmond]

☝️

[dungkhuc]

This is very useful as as it is data-diff would otherwise just stay stuck for many hours in our automation script if there are a big number of changes.

[glebmezh]

@JCZuurmond @dungkhuc ,

I'm sorry for the delay in following up on this. Thank you for taking the time to raise this issue!

We made a hard decision to sunset the data-diff package and won't provide further development or support.

If that's of interest, over the past few months, we have rewritten the diffing engine in Datafold Cloud and solved many issues that existed in this package. In particular, we implemented sampling, per-column diff limits, and real-time result updates to help with the problem you are describing.

-Gleb