Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1712

HAR sequence files throw errors in MR jobs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.1
    • Fix Version/s: None
    • Component/s: harchive
    • Labels:
      None

      Description

      When a HAR is specified as the input for a map reduce job and the file format is sequence file, an error similar to the following is thrown (this one is from Hive).

      java.lang.IllegalArgumentException: Offset 0 is outside of file (0..-1)
              at org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:299)
              at org.apache.hadoop.mapred.FileInputFormat.getSplitHosts(FileInputFormat.java:455)
              at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:260)
              at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:261)
              at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:827)
              at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:798)
              at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:747)
              at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:663)
              at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
              at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
              at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
              at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
              at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      

      This is caused by the dummy block location returned by HarFileSystem.getFileBlockLocations().

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mahadev Mahadev Konar
                Reporter:
                pauly Paul Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: