support merging primitive dictionaries in interleave and concat

[davidhewitt]

Which issue does this PR close?

• Closes interleave panics with dictionary key overflows if combined values arrays are longer than key size #7466

Rationale for this change

Fixes a panic :)

What changes are included in this PR?

Introduces a strategy for merging primitive dictionaries to deduplicate values when naive concatentation would overflow the key type.

Tests for both interleave and concat are included.

Are there any user-facing changes?

Just the bugfix.

[davidhewitt]

Suggested change

	"interleave does not yet support merging dictionaries with value type {other:?}"
	"concat does not yet support merging dictionaries with value type {other:?}"

[davidhewitt]

Changing this line allows should_merge_dictionary_values to return true for the primitive data types, but it also introduces a regression for weird types like dictionaries-of-unions. This heuristic might ask for these to merge if the values exceed the expected output length, which will now fail. Before they might be able to concat / interleave via the fallback methods even if merging is not supported (however they could easily hit the panic case).

I think probably the solution is to adjust this heuristic so it outputs why it needs to merge (e.g. must merge to attempt to avoid overflow) - and in callers, for overflow we would fail gracefully for types which can't yet be merged.

[davidhewitt]

I'll rework to fixup, please don't bother reviewing for the moment.

[davidhewitt]

Ok, this should be ready for review.

[tustvold]

If I am not mistaken this will generate non-trivial code for all possible values of dictionary key types and primitive key type, I don't think we can really do this. Perhaps you could look at how the arithmetic kernels, comparison kernels of other selection kernels avoid this.

For reference on my computer before this change a clean build of arrow-select and all dependents takes

$ time cargo build --release
Executed in    7.86 secs    fish           external
   usr time   40.93 secs    0.00 micros   40.93 secs
   sys time    1.55 secs  935.00 micros    1.54 secs

-- Second Run --
________________________________________________________
Executed in    7.80 secs    fish           external
   usr time   40.79 secs    1.10 millis   40.79 secs
   sys time    1.51 secs    0.03 millis    1.51 secs

Whereas this PR

________________________________________________________
Executed in   11.58 secs    fish           external
   usr time   57.66 secs    0.00 micros   57.66 secs
   sys time    1.72 secs  923.00 micros    1.71 secs

-- Second Run --
________________________________________________________
Executed in   15.21 secs    fish           external
   usr time   71.19 secs    1.07 millis   71.19 secs
   sys time    2.41 secs    0.00 millis    2.41 secs

Given how niche primitive dictionaries are, and how fundamental arrow-select is, we need to be very careful here.

[davidhewitt]

Great point, I have a few other tasks to handle in the coming days so I will seek to return to this soon, though it might be the end of the week or longer.