Hive
  1. Hive
  2. HIVE-3218

Stream table of SMBJoin/BucketMapJoin with two or more partitions is not handled properly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      drop table hive_test_smb_bucket1;
      drop table hive_test_smb_bucket2;
      
      create table hive_test_smb_bucket1 (key int, value string) partitioned by (ds string) clustered by (key) sorted by (key) into 2 buckets;
      create table hive_test_smb_bucket2 (key int, value string) partitioned by (ds string) clustered by (key) sorted by (key) into 2 buckets;
      
      set hive.enforce.bucketing = true;
      set hive.enforce.sorting = true;
      
      insert overwrite table hive_test_smb_bucket1 partition (ds='2010-10-14') select key, value from src;
      insert overwrite table hive_test_smb_bucket1 partition (ds='2010-10-15') select key, value from src;
      insert overwrite table hive_test_smb_bucket2 partition (ds='2010-10-15') select key, value from src;
      
      
      set hive.optimize.bucketmapjoin = true;
      set hive.optimize.bucketmapjoin.sortedmerge = true;
      set hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
      
      SELECT /* + MAPJOIN(b) */ * FROM hive_test_smb_bucket1 a JOIN hive_test_smb_bucket2 b ON a.key = b.key;
      

      which make bucket join context..

      Alias Bucket Output File Name Mapping:
              hdfs://localhost:9000/user/hive/warehouse/hive_test_smb_bucket1/ds=2010-10-14/000000_0 0
              hdfs://localhost:9000/user/hive/warehouse/hive_test_smb_bucket1/ds=2010-10-14/000001_0 1
              hdfs://localhost:9000/user/hive/warehouse/hive_test_smb_bucket1/ds=2010-10-15/000000_0 0
              hdfs://localhost:9000/user/hive/warehouse/hive_test_smb_bucket1/ds=2010-10-15/000001_0 1
      

      fails with exception

      java.lang.RuntimeException: Hive Runtime Error while closing operators
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:416)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
      	at org.apache.hadoop.mapred.Child.main(Child.java:264)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://localhost:9000/tmp/hive-navis/hive_2012-06-29_22-17-49_574_6018646381714861925/_task_tmp.-ext-10001/_tmp.000001_0 to: hdfs://localhost:9000/tmp/hive-navis/hive_2012-06-29_22-17-49_574_6018646381714861925/_tmp.-ext-10001/000001_0
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:198)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:100)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:717)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:557)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
      	at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
      	... 8 more
      
      1. hive.3218.2.patch
        166 kB
        Namit Jain
      2. HIVE-3218.1.patch.txt
        142 kB
        Navis

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development