Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6452

NPE when intermediate encrypt enabled for LocalRunner

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Enable the below properties try running mapreduce job

      mapreduce.framework.name=local
      mapreduce.job.encrypted-intermediate-data=true

      2015-08-14 16:27:25,248 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(561)) - job_local473843898_0001
      java.lang.Exception: java.lang.NullPointerException
              at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
              at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.crypto.CryptoOutputStream.<init>(CryptoOutputStream.java:92)
              at org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.<init>(CryptoFSDataOutputStream.java:31)
              at org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
              at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
              at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
              at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
              at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      Jobs are failing always

        Attachments

        1. TestLocalJobSubmission.java
          4 kB
          zhihai xu
        2. MAPREDUCE-6452.1.patch
          1 kB
          Ajith S
        3. MAPREDUCE-6452.002.patch
          5 kB
          zhihai xu

          Activity

            People

            • Assignee:
              zxu zhihai xu
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: