Hadoop Common
  1. Hadoop Common
  2. HADOOP-3299

org.apache.hadoop.mapred.join.CompositeInputFormat does not initialize TextInput format files with the configuration resulting in an NullPointerException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.3
    • Fix Version/s: 0.18.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it.

      Description

      The input formats are not initialized with the Configuration object before the isSplitable method is called.

      bin/hadoop jar hadoop-0.16.3-examples.jar join -r 1 -inFormat org.apache.hadoop.mapred.KeyValueTextInputFormat -outFormat org.apache.hadoop.mapred.TextOutputFormat -joinOp outer datajoin/input datajoin/output -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.mapred.join.TupleWritable
      08/04/22 15:05:33 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
      Job started: Tue Apr 22 15:05:33 GMT-08:00 2008
      08/04/22 15:05:33 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
      08/04/22 15:05:33 INFO mapred.FileInputFormat: Total input paths to process : 2
      java.lang.NullPointerException
      at org.apache.hadoop.mapred.KeyValueTextInputFormat.isSplitable(KeyValueTextInputFormat.java:44)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:185)
      at org.apache.hadoop.mapred.join.Parser$WNode.getSplits(Parser.java:304)
      at org.apache.hadoop.mapred.join.Parser$CNode.getSplits(Parser.java:374)
      at org.apache.hadoop.mapred.join.CompositeInputFormat.getSplits(CompositeInputFormat.java:129)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:542)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:803)
      at org.apache.hadoop.examples.Join.run(Join.java:149)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.examples.Join.main(Join.java:158)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
      at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
      at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:52)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:155)

      1. text.diff
        1 kB
        Jason
      2. keyvaluetext.diff
        1 kB
        Jason
      3. 3299-0.patch
        7 kB
        Chris Douglas

        Activity

        Jason created issue -
        Jason made changes -
        Field Original Value New Value
        Attachment text.diff [ 12380737 ]
        Jason made changes -
        Attachment keyvaluetext.diff [ 12380738 ]
        Chris Douglas made changes -
        Attachment 3299-0.patch [ 12380974 ]
        Chris Douglas made changes -
        Fix Version/s 0.18.0 [ 12312972 ]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Douglas made changes -
        Assignee Chris Douglas [ chris.douglas ]
        Owen O'Malley made changes -
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Robert Chansler made changes -
        Release Note Force the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it. Changed the TextInputFormat and KeyValueTextInput classes to initialize the compressionCodecs member variable before dereferencing it.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Jason
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development