Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18143

LazySimpleSerDe.LazyMap doesn't handle the condition of when map key is another column value

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.14.0, 1.2.2, 2.3.0
    • Fix Version/s: 2.3.0
    • Labels:
      None
    • Target Version/s:

      Description

      LazySimpleSerDe.LazyMap doesn't handle the condition of when map key is another column value.
      eg:

      CREATE TABLE `demo`(
      `create_timestamp` bigint,
      `ctime` string,
      `event_data` map<string,string>
      )
      ROW FORMAT SERDE
      'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
      STORED AS INPUTFORMAT
      'org.apache.hadoop.mapred.TextInputFormat'
      OUTPUTFORMAT
      'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

      add data
      INSERT INTO TABLE demo
      select
      1511192567909,
      '1511192567909',
      map('1511192567909', '05020918127a82816437ba9edac17fca42601');

      Do query:
      SELECT
      create_timestamp,
      ctime,
      event_data,
      event_data[create_timestamp],
      event_data[ctime]
      FROM demo;

      RES:

      create_timestamp ctime event_data _c3 _c4

      --------------------------------------------------------------------------------------------------------------------------------------+

      1511192567909 1511192567909 {"1511192567909":"05020918127a82816437ba9edac17fca42601"} 05020918127a82816437ba9edac17fca42601 NULL

      event_data[ctime] get value NULL
      _______________________________________

      In LazyMap.getMapValueElement method.

      // Some comments here
      public Object getMapValueElement(Object key) {
          if (!parsed) {
            parse();
          }
          // search for the key
          for (int i = 0; i < mapSize; i++) {
            LazyPrimitive<?, ?> lazyKeyI = uncheckedGetKey(i);
            if (lazyKeyI == null) {
              continue;
            }
            // getWritableObject() will convert LazyPrimitive to actual primitive
            // writable objects.
            Object keyI = lazyKeyI.getWritableObject();
            if (keyI == null) {
              continue;
            }
            if (keyI.equals(key)) {
              // Got a match, return the value
              return uncheckedGetValue(i);
            }
          }
          return null;
        }
      

      When the key is a cloumn and don't need convert, the key will be a LazyPrimitive object.
      keyI = lazyKeyI.getWritableObject() is actual primitive writable objects.
      keyI.equals(key) will be false.

        Attachments

          Activity

            People

            • Assignee:
              wei.wei wei
              Reporter:
              wei.wei wei
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: