Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2564

NullPointerException in WritableComparator

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Invalid
    • Affects Version/s: 0.20.203.0
    • Fix Version/s: 0.20.203.0
    • Component/s: task
    • Labels:
      None
    • Environment:

      java version "1.6.0_23"
      Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
      Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)

      hadoop-0.20.203.0rc1

      Description

      java.lang.NullPointerException
      at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
      at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
      at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
      at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

      It is easy to see why this is happening. The WritableComparator is created in JobConf line 776:

      JobConf.java
         return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
      }
      

      which calls

      WritableComparator.java
      protected WritableComparator(Class<? extends WritableComparable> keyClass) {
          this(keyClass, false);
        }
      
        protected WritableComparator(Class<? extends WritableComparable> keyClass,
            boolean createInstances) {
          this.keyClass = keyClass;
          if (createInstances) {
            key1 = newKey();
            key2 = newKey();
            buffer = new DataInputBuffer();
          } else {
            key1 = key2 = null;
            buffer = null;
          }
        }
      

      key1, key2, and buffer end up being null. When compare() is called the NPE is thrown because buffer is null

      WritableComparator.java
       public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
          try {
            buffer.reset(b1, s1, l1);                   // parse key1
            key1.readFields(buffer);
            
            buffer.reset(b2, s2, l2);                   // parse key2
            key2.readFields(buffer);
            
          } catch (IOException e) {
            throw new RuntimeException(e);
          }
          
          return compare(key1, key2);                   // compare them
        }
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jks Joseph Shraibman
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: