[SPARK-48019] ColumnVectors with dictionaries and nulls are not read/copied correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.3
• Fix Version/s: 4.0.0, 3.5.2, 3.4.4
• Component/s: SQL
• Labels: correctness pull-request-available
• Assignee: Gene Pang
• Reporter: Gene Pang
• Created: 27/Apr/24 01:41
• Updated: 04/Sep/24 21:06
• Resolved: 28/Apr/24 03:10

Description

ColumnVectors have APIs like getInts, getFloats and so on. Those return a primitive array with the contents of the vector. When the ColumnVector has a dictionary, the values are decoded with the dictionary before filling in the primitive array.

However, ColumnVectors can have nulls, and for those null entries, the dictionary id is irrelevant, and can also be invalid. The dictionary should not be used for the null entries of the vector. Sometimes, this can cause an ArrayIndexOutOfBoundsException .

In addition to the possible Exception, copying a ColumnarArray is not correct. A ColumnarArray contains a ColumnVector so it can contain null values. However, the copy() for primitive types does not take into account the null-ness of the entries, and blindly copies all the primitive values. That means the null entries get lost.

Issue Links

• links to: GitHub Pull Request #46254
• links to: GitHub Pull Request #46372