[HIVE-18143] LazySimpleSerDe.LazyMap doesn't handle the condition of when map key is another column value

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.14.0, 1.2.2, 2.3.0
• Component/s: Serializers/Deserializers
• Target Version/s: 2.3.0
• Assignee: wei
• Reporter: wei
• Created: 24/Nov/17 08:14
• Updated: 21/Oct/22 07:31

Description

LazySimpleSerDe.LazyMap doesn't handle the condition of when map key is another column value.
eg:

CREATE TABLE `demo`(
`create_timestamp` bigint,
`ctime` string,
`event_data` map<string,string>
)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

add data
INSERT INTO TABLE demo
select
1511192567909,
'1511192567909',
map('1511192567909', '05020918127a82816437ba9edac17fca42601');

Do query:
SELECT
create_timestamp,
ctime,
event_data,
event_data[create_timestamp],
event_data[ctime]
FROM demo;

RES：

create_timestamp

ctime

event_data

_c3

_c4

--------------------------------------------------------------------------------------------------------------------------------------+

1511192567909

1511192567909

{"1511192567909":"05020918127a82816437ba9edac17fca42601"}

05020918127a82816437ba9edac17fca42601

NULL

event_data[ctime] get value NULL
_______________________________________

In LazyMap.getMapValueElement method.

// Some comments here
public Object getMapValueElement(Object key) {
    if (!parsed) {
      parse();
    }
    // search for the key
    for (int i = 0; i < mapSize; i++) {
      LazyPrimitive<?, ?> lazyKeyI = uncheckedGetKey(i);
      if (lazyKeyI == null) {
        continue;
      }
      // getWritableObject() will convert LazyPrimitive to actual primitive
      // writable objects.
      Object keyI = lazyKeyI.getWritableObject();
      if (keyI == null) {
        continue;
      }
      if (keyI.equals(key)) {
        // Got a match, return the value
        return uncheckedGetValue(i);
      }
    }
    return null;
  }

When the key is a cloumn and don't need convert, the key will be a LazyPrimitive object.
keyI = lazyKeyI.getWritableObject() is actual primitive writable objects.
keyI.equals(key) will be false.