[IMPALA-9939] Fix Hive interop for HLL with STRING types

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: Not Applicable
• Component/s: Backend
• Epic Link: unlabelled-IMPALA-9593
• Epic Color: ghx-label-11
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 09/Jul/20 17:17
• Updated: 29/Jul/20 07:24
• Resolved: 29/Jul/20 07:24

Description

It turned out that Impala hashes STRINGs differently than Hive.

Impala's implementation simply hashes the original byte array (e.g. a UTF-8 encoded string), while Hive hashes the UTF-16 encoded char array behind java strings. If the STRING is cast to BINARY in Hive (e.g. ds_hll_sketch(cast(s as binary)) ), then it is interoperable with Impala's current implementation.

I am not sure how to proceed - we could UTF-16 encode the strings in Impala before hashing, but this would be pretty slow, and I think that Hive actually could be also faster if it would hash UTF-8 arrays - as STRINGs are stored as org.apache.hadoop.io.Text, they are currently UTF-8 decoded to java string first and could be hashed directly without any conversion. This would break compatibility with existing Hive produced sketches though.