fix: extend limit on number of regular arrays to concatenate

[ianna]

No description provided.

[ianna]

@jpivarski - I think, adding a few specializations to the kernels fixes the issue.

[jpivarski]

It would solve the issue without changing the implementation. We're still not allowing UnionArrays to have more than 127 types, right? Is the idea that we keep the axis != 0 RegularArray concatenation algorithm the way it is, allow bigger integer types in UnionArray.simplified, but raise an error if UnionArray.simplifed returns a UnionArray? (Which it shouldn't even be able to build because UnionArray constructors should continue to require tags to be int8.)

[ianna]

It would solve the issue without changing the implementation. We're still not allowing UnionArrays to have more than 127 types, right? Is the idea that we keep the axis != 0 RegularArray concatenation algorithm the way it is, allow bigger integer types in UnionArray.simplified, but raise an error if UnionArray.simplifed returns a UnionArray? (Which it shouldn't even be able to build because UnionArray constructors should continue to require tags to be int8.)

These kernels are only used to make an index in:

awkward/src/awkward/operations/ak_concatenate.py

Line 278 in 4db7973

index = ak.contents.UnionArray.regular_index(tags, backend=backend)

Then this index is used to create a simplified "UnionArray" and return it as a RegularArray:

awkward/src/awkward/operations/ak_concatenate.py

Lines 278 to 288 in 4db7973

	index = ak.contents.UnionArray.regular_index(tags, backend=backend)
	inner = ak.contents.UnionArray.simplified(
	tags,
	index,
	[x._content for x in regulararrays],
	mergebool=mergebool,
	)
	return (ak.contents.RegularArray(inner, prototype.size),)
	elif all(

[jpivarski]

Okay, this is fine—the helper functions were designed for UnionArrays, but here they're being used for concatenation and not UnionArrays.

Do you really need int32, uint32, and int64 versions? If it's just an intermediate step in the ak.concatenate operation, why not make that intermediate step always int64? In general, the reason we have multiple dtypes of these nodes is to accept data with those dtypes from other sources without a copy-transformation. In fact, UnionArray's uint32 was motivated by a very early version of RNTuple—before that, Awkward didn't have any unsigned index types. 32-bit indexes in general were introduced for the sake of Arrow (along with BitMaskedArray).

[ianna]

Okay, this is fine—the helper functions were designed for UnionArrays, but here they're being used for concatenation and not UnionArrays.

Do you really need int32, uint32, and int64 versions? If it's just an intermediate step in the ak.concatenate operation, why not make that intermediate step always int64? In general, the reason we have multiple dtypes of these nodes is to accept data with those dtypes from other sources without a copy-transformation. In fact, UnionArray's uint32 was motivated by a very early version of RNTuple—before that, Awkward didn't have any unsigned index types. 32-bit indexes in general were introduced for the sake of Arrow (along with BitMaskedArray).

Yes, initially I thought we do not need all of them. However, looking at the code again it seems, yes, we need all of them: these are specializations for the tags defined as either int8 or int64 type.