Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6452

NPE when intermediate encrypt enabled for LocalRunner

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Enable the below properties try running mapreduce job

      mapreduce.framework.name=local
      mapreduce.job.encrypted-intermediate-data=true

      2015-08-14 16:27:25,248 WARN  [Thread-21] mapred.LocalJobRunner (LocalJobRunner.java:run(561)) - job_local473843898_0001
      java.lang.Exception: java.lang.NullPointerException
              at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
              at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.crypto.CryptoOutputStream.<init>(CryptoOutputStream.java:92)
              at org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.<init>(CryptoFSDataOutputStream.java:31)
              at org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
              at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
              at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
              at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
              at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      Jobs are failing always

      1. TestLocalJobSubmission.java
        4 kB
        zhihai xu
      2. MAPREDUCE-6452.1.patch
        1 kB
        Ajith S
      3. MAPREDUCE-6452.002.patch
        5 kB
        zhihai xu

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2248 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2248/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2248 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2248/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #310 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/310/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #310 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/310/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2267/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2267 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2267/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1051 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1051/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1051 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1051/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #323 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/323/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #323 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/323/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8365 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8365/)
        MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8365 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8365/ ) MAPREDUCE-6452 . NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu (zxu: rev cbb249534aa72ff6c290c4f99766415aeea9d6f5) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
        Hide
        zxu zhihai xu added a comment -

        thanks Arun Suresh, Ajith S and Mohammad Shahid Khan for the review! thanks Bibin A Chundatt for reporting this issue! Committed it to 2.8.0 and branch-2

        Show
        zxu zhihai xu added a comment - thanks Arun Suresh , Ajith S and Mohammad Shahid Khan for the review! thanks Bibin A Chundatt for reporting this issue! Committed it to 2.8.0 and branch-2
        Hide
        zxu zhihai xu added a comment -

        thanks for the review Arun Suresh, Ajith S and Mohammad Shahid Khan, All these test failures are not related to the patch. All these test passed at my local build.

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.091 sec - in org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
        Results :
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.402 sec - in org.apache.hadoop.mapreduce.security.TestJHSSecurity\
        Results :
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.584 sec - in org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
        Results :
        Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
        

        If no objection, will commit it tomorrow.

        Show
        zxu zhihai xu added a comment - thanks for the review Arun Suresh , Ajith S and Mohammad Shahid Khan , All these test failures are not related to the patch. All these test passed at my local build. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.091 sec - in org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.mapreduce.security.TestJHSSecurity Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.402 sec - in org.apache.hadoop.mapreduce.security.TestJHSSecurity\ Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.mapreduce.security.TestBinaryTokenFile Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.584 sec - in org.apache.hadoop.mapreduce.security.TestBinaryTokenFile Results : Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 If no objection, will commit it tomorrow.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 37s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 49s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 9s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common.
        -1 mapreduce tests 103m 27s Tests failed in hadoop-mapreduce-client-jobclient.
            143m 57s  



        Reason Tests
        Failed unit tests hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
          hadoop.mapreduce.security.TestJHSSecurity
          hadoop.mapreduce.security.TestBinaryTokenFile



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12752386/MAPREDUCE-6452.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a4d9acc
        hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 49s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 9s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 59s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common. -1 mapreduce tests 103m 27s Tests failed in hadoop-mapreduce-client-jobclient.     143m 57s   Reason Tests Failed unit tests hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken   hadoop.mapreduce.security.TestJHSSecurity   hadoop.mapreduce.security.TestBinaryTokenFile Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752386/MAPREDUCE-6452.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5954/console This message was automatically generated.
        Hide
        ajithshetty Ajith S added a comment -

        can be overlooked, i am still ok with your patch without this suggestion

        Show
        ajithshetty Ajith S added a comment - can be overlooked, i am still ok with your patch without this suggestion
        Hide
        ajithshetty Ajith S added a comment -

        Hi zhihai xu

        Patch looks great, a small suggestion, currently i see ENC_SPILL_KEY being also set at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initJobCredentialsAndUGI(Configuration) so can we push this and your changes to a common location like org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(Job, Cluster)

        Show
        ajithshetty Ajith S added a comment - Hi zhihai xu Patch looks great, a small suggestion, currently i see ENC_SPILL_KEY being also set at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initJobCredentialsAndUGI(Configuration) so can we push this and your changes to a common location like org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(Job, Cluster)
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for the patch zhihai xu and good catch Bibin A Chundatt.
        +1 pending jenkins..

        Show
        asuresh Arun Suresh added a comment - Thanks for the patch zhihai xu and good catch Bibin A Chundatt . +1 pending jenkins..
        Hide
        mohdshahidkhan Mohammad Shahid Khan added a comment -

        +1 for zhihai xu patch for the branch 2.7.1 , trunk and branch-2.

        Show
        mohdshahidkhan Mohammad Shahid Khan added a comment - +1 for zhihai xu patch for the branch 2.7.1 , trunk and branch-2.
        Hide
        zxu zhihai xu added a comment -

        I think this issue will depend on the commit "Fixing MR intermediate spills.":
        https://github.com/apache/hadoop/commit/6b710a42e00acca405e085724c89cda016cf7442
        Without the change from "Fixing MR intermediate spills.", the patch for this issue may not work correctly.

        Show
        zxu zhihai xu added a comment - I think this issue will depend on the commit "Fixing MR intermediate spills.": https://github.com/apache/hadoop/commit/6b710a42e00acca405e085724c89cda016cf7442 Without the change from "Fixing MR intermediate spills.", the patch for this issue may not work correctly.
        Hide
        zxu zhihai xu added a comment -

        thanks Ajith S and Mohammad Shahid Khan for the clarification, I see the commit "Fixing MR intermediate spills.": https://github.com/apache/hadoop/commit/6b710a42e00acca405e085724c89cda016cf7442
        change:

        private static byte[] getEncryptionKey() throws IOException {
            return TokenCache. getShuffleSecretKey(UserGroupInformation.getCurrentUser()
                    .getCredentials());
          }
        

        To

        private static byte[] getEncryptionKey() throws IOException {
            return TokenCache.getEncryptedSpillKey(UserGroupInformation.getCurrentUser()
                    .getCredentials());
          }
        

        The change "Fixing MR intermediate spills." is added at 2.7.1 release. But the stack trace from this JIRA is based on the code 2.7.1 or later
        because

        at org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
        

        doesn't match 2.7.0 code base.

        Show
        zxu zhihai xu added a comment - thanks Ajith S and Mohammad Shahid Khan for the clarification, I see the commit "Fixing MR intermediate spills.": https://github.com/apache/hadoop/commit/6b710a42e00acca405e085724c89cda016cf7442 change: private static byte [] getEncryptionKey() throws IOException { return TokenCache. getShuffleSecretKey(UserGroupInformation.getCurrentUser() .getCredentials()); } To private static byte [] getEncryptionKey() throws IOException { return TokenCache.getEncryptedSpillKey(UserGroupInformation.getCurrentUser() .getCredentials()); } The change "Fixing MR intermediate spills." is added at 2.7.1 release. But the stack trace from this JIRA is based on the code 2.7.1 or later because at org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112) doesn't match 2.7.0 code base.
        Hide
        mohdshahidkhan Mohammad Shahid Khan added a comment -

        Agree with Ajith S, even i have noticed the same thing.

        Show
        mohdshahidkhan Mohammad Shahid Khan added a comment - Agree with Ajith S, even i have noticed the same thing.
        Hide
        ajithshetty Ajith S added a comment -

        Hi zhihai xu

        Yes, i wanted to say that secretKeysMap set in a credential object, is not propagated. In my scenario, i have getSecretKey(credentials, SHUFFLE_TOKEN) returned null, because the map is empty.
        The reason my patch is failing your test case is
        In hadoop < 2.7.1, the org.apache.hadoop.mapreduce.CryptoUtils.getEncryptionKey() depends on SHUFFLE_TOKEN, so you patch will have problem in that version and in hadoop >= 2.7.1 it depends on ENC_SPILL_KEY as you said.

        I had patched it for a older version, I see that you have added a test case, feel free to assign to yourself

        Show
        ajithshetty Ajith S added a comment - Hi zhihai xu Yes, i wanted to say that secretKeysMap set in a credential object, is not propagated. In my scenario, i have getSecretKey(credentials, SHUFFLE_TOKEN) returned null, because the map is empty. The reason my patch is failing your test case is In hadoop < 2.7.1, the org.apache.hadoop.mapreduce.CryptoUtils.getEncryptionKey() depends on SHUFFLE_TOKEN, so you patch will have problem in that version and in hadoop >= 2.7.1 it depends on ENC_SPILL_KEY as you said. I had patched it for a older version, I see that you have added a test case, feel free to assign to yourself
        Hide
        zxu zhihai xu added a comment -

        Thanks for working on this issue Ajith S! I verified your patch, It doesn't work. I attached the test I used for the verification.
        The culprit of the issue is not missing credentials but missing SecretKey ENC_SPILL_KEY.
        UserGroupInformation.getCurrentUser().getCredentials() will never return null based on the following code.

          public Credentials getCredentials() {
            synchronized (subject) {
              Credentials creds = new Credentials(getCredentialsInternal());
              Iterator<Token<?>> iter = creds.getAllTokens().iterator();
              while (iter.hasNext()) {
                if (iter.next() instanceof Token.PrivateToken) {
                  iter.remove();
                }
              }
              return creds;
            }
          }
        

        Since credentials is not null in getSecretKey, credentials.getSecretKey(ENC_SPILL_KEY) must be null which triggered the NPE.

          private static byte[] getEncryptionKey() throws IOException {
            return TokenCache.getEncryptedSpillKey(UserGroupInformation.getCurrentUser()
                    .getCredentials());
          }
          public static byte[] getEncryptedSpillKey(Credentials credentials) {
            return getSecretKey(credentials, ENC_SPILL_KEY);
          }
          public static byte[] getSecretKey(Credentials credentials, Text alias) {
            if(credentials == null)
              return null;
            return credentials.getSecretKey(alias);
          }
        

        I attached a patch MAPREDUCE-6452.000.patch which should fix this issue. The patch includes a test case to reproduce this issue. It sets the SecretKey ENC_SPILL_KEY in the credentials of current user by calling TokenCache.setEncryptedSpillKey
        By the way, thanks Bibin A Chundatt for reporting this issue.

        Show
        zxu zhihai xu added a comment - Thanks for working on this issue Ajith S ! I verified your patch, It doesn't work. I attached the test I used for the verification. The culprit of the issue is not missing credentials but missing SecretKey ENC_SPILL_KEY . UserGroupInformation.getCurrentUser().getCredentials() will never return null based on the following code. public Credentials getCredentials() { synchronized (subject) { Credentials creds = new Credentials(getCredentialsInternal()); Iterator<Token<?>> iter = creds.getAllTokens().iterator(); while (iter.hasNext()) { if (iter.next() instanceof Token.PrivateToken) { iter.remove(); } } return creds; } } Since credentials is not null in getSecretKey , credentials.getSecretKey(ENC_SPILL_KEY) must be null which triggered the NPE. private static byte [] getEncryptionKey() throws IOException { return TokenCache.getEncryptedSpillKey(UserGroupInformation.getCurrentUser() .getCredentials()); } public static byte [] getEncryptedSpillKey(Credentials credentials) { return getSecretKey(credentials, ENC_SPILL_KEY); } public static byte [] getSecretKey(Credentials credentials, Text alias) { if (credentials == null ) return null ; return credentials.getSecretKey(alias); } I attached a patch MAPREDUCE-6452 .000.patch which should fix this issue. The patch includes a test case to reproduce this issue. It sets the SecretKey ENC_SPILL_KEY in the credentials of current user by calling TokenCache.setEncryptedSpillKey By the way, thanks Bibin A Chundatt for reporting this issue.
        Hide
        ajithshetty Ajith S added a comment -

        org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(Job, Cluster) adds a generated shuffle key in case the credentials are not set, in case of the LocalJobRunner, this need to be propogated. Fixed accordingly

        Show
        ajithshetty Ajith S added a comment - org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(Job, Cluster) adds a generated shuffle key in case the credentials are not set, in case of the LocalJobRunner, this need to be propogated. Fixed accordingly
        Hide
        ajithshetty Ajith S added a comment -

        Hi zhihai xu

        Sorry for delay, had already worked on it. Please let me know if you have a different approach

        Show
        ajithshetty Ajith S added a comment - Hi zhihai xu Sorry for delay, had already worked on it. Please let me know if you have a different approach
        Hide
        zxu zhihai xu added a comment -

        Hi Ajith S, Can I take this issue if you didn't start working on it yet? I have a patch ready for this issue. thanks

        Show
        zxu zhihai xu added a comment - Hi Ajith S , Can I take this issue if you didn't start working on it yet? I have a patch ready for this issue. thanks
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Allen Wittenauer Any thoughts on this issue?

        Show
        bibinchundatt Bibin A Chundatt added a comment - Allen Wittenauer Any thoughts on this issue?

          People

          • Assignee:
            zxu zhihai xu
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development