Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20962

CommonMergeJoinOperator cannot join on complex keys

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.4
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None

      Description

      CommonMergeJoinOperator fails to perform joins on complex keys, e.g.

       

      CREATE TABLE complex_key (
      `key` struct<id:bigint, country:stringvalue int)
      PARTITIONED BY (date int);
      
      SELECT t1.key, t1.value, t2.value FROM
      complex_key t1
      FULL OUTER JOIN
      complex_key t2
      ON (t1.date=20181121 and t2.date =20181122 AND t1.key=t2.key);
      

      This causes a ClassCastException:

      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=1) {"key":{"reducesinkkey0":{"id":1,"country":"DK"}},"value":{"_col0":1489}}
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
      	... 22 more
      Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.apache.hadoop.io.WritableComparable
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.compareKeys(CommonMergeJoinOperator.java:543)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.processKey(CommonMergeJoinOperator.java:516)
      	at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:212)
      	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
      

       
      Because the compareKeys() method tries to cast each key to a WritableComparable but e.g. the StandardStructObjectInspector would return our key field as an Arraylist.
      https://github.com/apache/hive/blob/66f97da9de65b1c7151ec57bdf9ada937855bd75/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java#L590

      Proper way to do it would probably be to use the KeyWrapperFactory to convert the keys to something easily comparable?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cvaliente Clemens Valiente
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: