Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17774

compaction may start with 0 splits and fail

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.4.0
    • Transactions
    • None

    Description

      2017-09-26 10:36:01,979 INFO  [...]: compactor.CompactorMR (CompactorMR.java:launchCompactionJob(295)) - 
      Submitting MINOR compaction job ....
       (current delta dirs count=0, obsolete delta dirs count=0. TxnIdRange[9223372036854775807,-9223372036854775808]
      ...
      2017-09-26 10:36:02,350 INFO  [...]: mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of splits:0
      ...
      2017-09-26 10:36:08,637 INFO  [...]: mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - 
      Job job_1503950256860_15982 failed with state FAILED due to: No of maps and reduces are 0 job_1503950256860_15982
      Job commit failed: java.io.FileNotFoundException: File .../hello_acid/load_date=2016-03-03/_tmp_a95346ad-bd89-4e66-9b05-e60fdfa11858 does not exist.
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
      	at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
      	at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
      	at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
      	at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Looks like the MR job should not have been attempted in this case.

      Attachments

        1. HIVE-17774-branch-2.patch
          0.8 kB
          Sergey Shelukhin

        Issue Links

          Activity

            People

              sershe Sergey Shelukhin
              sershe Sergey Shelukhin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: