Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3299

org.apache.hadoop.mapred.join.CompositeInputFormat does not initialize TextInput format files with the configuration resulting in an NullPointerException

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.3
    • Fix Version/s: 0.18.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it.

      Description

      The input formats are not initialized with the Configuration object before the isSplitable method is called.

      bin/hadoop jar hadoop-0.16.3-examples.jar join -r 1 -inFormat org.apache.hadoop.mapred.KeyValueTextInputFormat -outFormat org.apache.hadoop.mapred.TextOutputFormat -joinOp outer datajoin/input datajoin/output -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.mapred.join.TupleWritable
      08/04/22 15:05:33 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
      Job started: Tue Apr 22 15:05:33 GMT-08:00 2008
      08/04/22 15:05:33 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
      08/04/22 15:05:33 INFO mapred.FileInputFormat: Total input paths to process : 2
      java.lang.NullPointerException
      at org.apache.hadoop.mapred.KeyValueTextInputFormat.isSplitable(KeyValueTextInputFormat.java:44)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:185)
      at org.apache.hadoop.mapred.join.Parser$WNode.getSplits(Parser.java:304)
      at org.apache.hadoop.mapred.join.Parser$CNode.getSplits(Parser.java:374)
      at org.apache.hadoop.mapred.join.CompositeInputFormat.getSplits(CompositeInputFormat.java:129)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:542)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:803)
      at org.apache.hadoop.examples.Join.run(Join.java:149)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.Join.main(Join.java:158)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:52)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:155)

        Attachments

        1. text.diff
          1 kB
          Jason
        2. keyvaluetext.diff
          1 kB
          Jason
        3. 3299-0.patch
          7 kB
          Chris Douglas

          Activity

            People

            • Assignee:
              chris.douglas Chris Douglas
              Reporter:
              jason_attributor Jason
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: