[ARROW-4501] [C++] Unique returns non-unique strings

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.1, 0.13.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21053
• Assignee: Ilya Tokar
• Reporter: Ilya Tokar
• Created: 07/Feb/19 21:13
• Updated: 11/Jan/23 07:34
• Resolved: 11/Feb/19 18:39

Description

Calling Unique on  e. g. {"some long string data","some long string data","other data"} returns

dictionary with "some long string data" appearing twice. This is caused by off by 1 error in DoubleCrcHash, which caused it to read 1 byte past the end of the strings with length  higher than 16, and not divisible by 4. In such cases, we never hash p[0], and we always read one extra byte.

Issue Links

• links to: GitHub Pull Request #3584