Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4627

Total ordering of Hive output

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 0.11.0
    • None
    • None
    • None
    • I'm not able to reproduce this behavior. I now suspect the input data set was empty, causing this runtime exception.

    Description

      I'd like to use Hive to generate HFiles for HBase. I started off by following the instructions on the wiki, but that took me only so far. TotalOrderPartitioning didn't work. That took me to this post which points out that Hive partitions on value instead of key. A patched TOP brings me to this error:

      2013-05-17 21:00:47,781 WARN org.apache.hadoop.mapred.Child: Error running child
      java.lang.RuntimeException: Hive Runtime Error while closing operators: java.io.IOException: No files found in hdfs://ip-10-191-3-134.ec2.internal:8020/tmp/hive-hrt_qa/hive_2013-05-17_20-58-58_357_6896546413926013201/_task_tmp.-ext-10000/_tmp.000000_0
      	at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
      	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:532)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
      	at org.apache.hadoop.mapred.Child.main(Child.java:249)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: No files found in hdfs://ip-10-191-3-134.ec2.internal:8020/tmp/hive-hrt_qa/hive_2013-05-17_20-58-58_357_6896546413926013201/_task_tmp.-ext-10000/_tmp.000000_0
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:183)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:865)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
      	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
      	at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
      	... 7 more
      Caused by: java.io.IOException: No files found in hdfs://ip-10-191-3-134.ec2.internal:8020/tmp/hive-hrt_qa/hive_2013-05-17_20-58-58_357_6896546413926013201/_task_tmp.-ext-10000/_tmp.000000_0
      	at org.apache.hadoop.hive.hbase.HiveHFileOutputFormat$1.close(HiveHFileOutputFormat.java:142)
      	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:180)
      	... 11 more
      

      Attachments

        1. hive-partitioner.patch
          0.7 kB
          Nick Dimiduk
        2. 02_hfiles.hql
          0.4 kB
          Nick Dimiduk
        3. 01_sample.hql
          0.6 kB
          Nick Dimiduk
        4. 00_tables.ddl
          1 kB
          Nick Dimiduk

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            ndimiduk Nick Dimiduk
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment