Does the arrow protocol require unique dictionary values?

[kdkavanagh]

Is it explicitly required that dictionary values be unique? Currently the at least the C++ API DictionaryMemoTable will deduplicate entries on insert, however I can think of usecases where multiple underlying index values should map to the same dictionary value. Given the O(1) lookup from index->value, this would require tolerating duplicates in the dictionary array.

Example of creating non-unique dicts: #25845

Curious if this is a hard-requirement, or if it's an implementation detail open to change.

[felipecrv]

Code processing dictionary-encoded arrays should not assume values to be unique.

[assignUser]

From the Arrow format spec:

Note that a dictionary is permitted to contain duplicate values or nulls:

data VarBinary (dictionary-encoded)
   index_type: Int32
   values: [0, 1, 3, 1, 4, 2]

dictionary
   type: VarBinary
   values: ['foo', 'bar', 'baz', 'foo', null]

The null count of such arrays is dictated only by the validity bitmap of its indices, irrespective of any null values in the dictionary.
Arrow Columnar Format – Dictionary-encoded Layout

(Thanks Docs chat bot!)

So forcing the de-duplication seems to go against the spec and there for is a bug?

[raulcd]

Is there an issue opened to track here? Otherwise, @kdkavanagh can you open an issue, please?

[kdkavanagh]

Created #47151, thanks