Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.1
    • Component/s: client, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805.

      1. MAPREDUCE-5815.2.patch
        5 kB
        Akira Ajisaka
      2. MAPREDUCE-5815.3.patch
        6 kB
        Vinod Kumar Vavilapalli
      3. MAPREDUCE-5815.v01.patch
        1 kB
        Gera Shegalov

        Issue Links

          Activity

          Hide
          Gera Shegalov added a comment -

          v01 patch for review

          Show
          Gera Shegalov added a comment - v01 patch for review
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12637306/MAPREDUCE-5815.v01.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637306/MAPREDUCE-5815.v01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4466//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Gera Shegalov - I'm not sure we should quietly ignore the queue-name... maybe the right fix is to ensure TestMRAppMaster actually sets a proper queue name?

          Vinod Kumar Vavilapalli - Thoughts?

          Show
          Arun C Murthy added a comment - Gera Shegalov - I'm not sure we should quietly ignore the queue-name... maybe the right fix is to ensure TestMRAppMaster actually sets a proper queue name? Vinod Kumar Vavilapalli - Thoughts?
          Hide
          Mit Desai added a comment -

          I agree with Arun C Murthy that TestMRAppMaster should set a proper queue name.
          On the other note, instead of ignoring the queue name, I think it would be a good idea to have the null check and throw an appropriate exception. What do you guys think?

          Show
          Mit Desai added a comment - I agree with Arun C Murthy that TestMRAppMaster should set a proper queue name. On the other note, instead of ignoring the queue name, I think it would be a good idea to have the null check and throw an appropriate exception. What do you guys think?
          Hide
          Akira Ajisaka added a comment -

          I agree with you that TestMRAppMaster should set a proper queue name.

          I think it would be a good idea to have the null check and throw an appropriate exception.

          I suggest to handle null name as "NA" by using getNonEmptyString. The method is already used for user name and job name.

          Show
          Akira Ajisaka added a comment - I agree with you that TestMRAppMaster should set a proper queue name. I think it would be a good idea to have the null check and throw an appropriate exception. I suggest to handle null name as "NA" by using getNonEmptyString . The method is already used for user name and job name.
          Hide
          Akira Ajisaka added a comment -

          Attaching a patch to

          1. set a proper queue name when JobHistoryEventHandler instantiate JobIndexInfo.
          2. handle null queue name as "NA" by using FileIndexUtils#getNonEmptyString.
          Show
          Akira Ajisaka added a comment - Attaching a patch to set a proper queue name when JobHistoryEventHandler instantiate JobIndexInfo . handle null queue name as "NA" by using FileIndexUtils#getNonEmptyString .
          Hide
          Akira Ajisaka added a comment -

          Gera Shegalov, I created a patch to fix this issue since this jira had not been updated for several days. Can I take over this issue?

          Show
          Akira Ajisaka added a comment - Gera Shegalov , I created a patch to fix this issue since this jira had not been updated for several days. Can I take over this issue?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639163/MAPREDUCE-5815.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4490//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4490//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639163/MAPREDUCE-5815.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4490//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4490//console This message is automatically generated.
          Hide
          Gera Shegalov added a comment -

          Thanks, Akira Ajisaka. Yeah, I didn't have the bandwidth to deal with this JIRA now and I was waiting for Vinod Kumar Vavilapalli to chime in as requested by Arun C Murthy. Reassigning ...

          Show
          Gera Shegalov added a comment - Thanks, Akira Ajisaka . Yeah, I didn't have the bandwidth to deal with this JIRA now and I was waiting for Vinod Kumar Vavilapalli to chime in as requested by Arun C Murthy . Reassigning ...
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Sorry, didn't see this before. Reviewing now..

          Show
          Vinod Kumar Vavilapalli added a comment - Sorry, didn't see this before. Reviewing now..
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I think the real issue is that the AM on restart after a crash-during-comment doesn't write JOB_SUBMITTED events and so queue-name/user-name etc are missing. But that's lot more changes - to emit user-name/queue-name etc as part of AM_STARTED event that is always emmited. I'll file a separate ticket for that.

          The current patch seems mostly fine, except for the repeated usage of "default" strings. I'll change it to to JobConf.DEFAULT_QUEUE_NAME myself.

          Show
          Vinod Kumar Vavilapalli added a comment - I think the real issue is that the AM on restart after a crash-during-comment doesn't write JOB_SUBMITTED events and so queue-name/user-name etc are missing. But that's lot more changes - to emit user-name/queue-name etc as part of AM_STARTED event that is always emmited. I'll file a separate ticket for that. The current patch seems mostly fine, except for the repeated usage of "default" strings. I'll change it to to JobConf.DEFAULT_QUEUE_NAME myself.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Same patch as before but without the "default" strings.

          Will commit this if Jenkins says okay.

          Show
          Vinod Kumar Vavilapalli added a comment - Same patch as before but without the "default" strings. Will commit this if Jenkins says okay.
          Hide
          Akira Ajisaka added a comment -

          Thanks Vinod for reviewing and updating the patch!

          Show
          Akira Ajisaka added a comment - Thanks Vinod for reviewing and updating the patch!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12639696/MAPREDUCE-5815.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4500//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4500//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639696/MAPREDUCE-5815.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4500//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4500//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I think the real issue is that the AM on restart after a crash-during-comment doesn't write JOB_SUBMITTED events and so queue-name/user-name etc are missing. But that's lot more changes - to emit user-name/queue-name etc as part of AM_STARTED event that is always emmited. I'll file a separate ticket for that.

          Filed MAPREDUCE-5829.

          Checking this in now.

          Show
          Vinod Kumar Vavilapalli added a comment - I think the real issue is that the AM on restart after a crash-during-comment doesn't write JOB_SUBMITTED events and so queue-name/user-name etc are missing. But that's lot more changes - to emit user-name/queue-name etc as part of AM_STARTED event that is always emmited. I'll file a separate ticket for that. Filed MAPREDUCE-5829 . Checking this in now.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk, branch-2 and branch-2.4. Thanks Akira!

          Show
          Vinod Kumar Vavilapalli added a comment - Committed this to trunk, branch-2 and branch-2.4. Thanks Akira!
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5497/)
          MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5497/ ) MAPREDUCE-5815 . Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #537 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/537/)
          MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #537 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/537/ ) MAPREDUCE-5815 . Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1729 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1729/)
          MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1729 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1729/ ) MAPREDUCE-5815 . Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1754/)
          MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1754/ ) MAPREDUCE-5815 . Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1586559 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java

            People

            • Assignee:
              Akira Ajisaka
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development