Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: mr-am, mrv2
    • Labels:

      Description

      While running the shutdown hook of MRAppMaster, hit NPE

      Exception in thread "Thread-1" java.lang.NullPointerException
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
      
      1. MAPREDUCE-4071-2.patch
        9 kB
        Bhallamudi Venkata Siva Kamesh
      2. MAPREDUCE-4071-2.patch
        9 kB
        Bhallamudi Venkata Siva Kamesh
      3. MAPREDUCE-4071-1.patch
        6 kB
        Bhallamudi Venkata Siva Kamesh
      4. MAPREDUCE-4071.patch
        1 kB
        Bhallamudi Venkata Siva Kamesh

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528570/MAPREDUCE-4071-2.patch
        against trunk revision 47f7f18.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5259//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528570/MAPREDUCE-4071-2.patch against trunk revision 47f7f18. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5259//console This message is automatically generated.
        Hide
        airbots Chen He added a comment -

        ping

        Show
        airbots Chen He added a comment - ping
        Hide
        nemon Nemon Lou added a comment -

        When split metadata size exceeds 10000000,MRAppMaster will throw an IOException during start and the NPE shows up.
        Bhallamudi,are you still working on this?

        Show
        nemon Nemon Lou added a comment - When split metadata size exceeds 10000000,MRAppMaster will throw an IOException during start and the NPE shows up. Bhallamudi,are you still working on this?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528570/MAPREDUCE-4071-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

        org.apache.hadoop.mapreduce.v2.app.TestRecovery
        org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2410//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2410//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528570/MAPREDUCE-4071-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2410//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2410//console This message is automatically generated.
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        Kicking Jenkins again

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - Kicking Jenkins again
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        Submitting the same patch again

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - Submitting the same patch again
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522570/MAPREDUCE-4071-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
        org.apache.hadoop.yarn.server.TestDiskFailures
        org.apache.hadoop.yarn.server.TestContainerManagerSecurity
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
        org.apache.hadoop.mapreduce.v2.app.TestRecovery
        org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
        org.apache.hadoop.mapred.TestMiniMRClasspath
        org.apache.hadoop.mapreduce.v2.TestMRJobs
        org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
        org.apache.hadoop.mapred.TestMiniMRBringup
        org.apache.hadoop.mapred.TestMiniMRChildTask
        org.apache.hadoop.mapred.TestReduceFetch
        org.apache.hadoop.mapred.TestClusterMRNotification
        org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
        org.apache.hadoop.mapred.TestJobCounters
        org.apache.hadoop.mapreduce.TestChild
        org.apache.hadoop.mapred.TestMiniMRClientCluster
        org.apache.hadoop.ipc.TestSocketFactory
        org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
        org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
        org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
        org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
        org.apache.hadoop.mapred.TestClientRedirect
        org.apache.hadoop.mapred.TestLazyOutput
        org.apache.hadoop.mapred.TestJobCleanup
        org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
        org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
        org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
        org.apache.hadoop.conf.TestNoDefaultsJobConf
        org.apache.hadoop.mapreduce.v2.TestRMNMInfo
        org.apache.hadoop.mapred.TestClusterMapReduceTestCase
        org.apache.hadoop.mapreduce.v2.TestNonExistentJob
        org.apache.hadoop.mapred.TestJobSysDirWithDFS
        org.apache.hadoop.mapreduce.v2.TestUberAM
        org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
        org.apache.hadoop.mapred.TestJobName
        org.apache.hadoop.mapreduce.security.TestJHSSecurity

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2220//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2220//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522570/MAPREDUCE-4071-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.server.TestDiskFailures org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2220//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2220//console This message is automatically generated.
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        To fix this issue, I just re-factored the existing code. After re-factoring I successfully ran wordcount and terasort examples.

        Please review the patch and provide your comments.

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - To fix this issue, I just re-factored the existing code. After re-factoring I successfully ran wordcount and terasort examples. Please review the patch and provide your comments.
        Hide
        revans2 Robert Joseph Evans added a comment -

        The patch does not appear to compile against trunk.

        /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java:[1217,4] method does not override or implement a method from a supertype
        

        It could be a hickup in Jenkins, but could you please try to upmerge and verify that everything still builds and runs correctly.

        Show
        revans2 Robert Joseph Evans added a comment - The patch does not appear to compile against trunk. /home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java:[1217,4] method does not override or implement a method from a supertype It could be a hickup in Jenkins, but could you please try to upmerge and verify that everything still builds and runs correctly.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521491/MAPREDUCE-4071-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause tar ant target to fail.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521491/MAPREDUCE-4071-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//console This message is automatically generated.
        Hide
        revans2 Robert Joseph Evans added a comment -

        Kicking Jenkins again

        Show
        revans2 Robert Joseph Evans added a comment - Kicking Jenkins again
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        Hi Robert,
        Thanks for looking into the patch. I totally ignored the unregister method in RMCommunicator. I modified the patch and most of the changes are re-factoring like moving object instantiation to init method etc... If you feel, this is not the right way to fix this, please look into this issue.

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - Hi Robert, Thanks for looking into the patch. I totally ignored the unregister method in RMCommunicator. I modified the patch and most of the changes are re-factoring like moving object instantiation to init method etc... If you feel, this is not the right way to fix this, please look into this issue.
        Hide
        revans2 Robert Joseph Evans added a comment -

        Can you please address the findbugs warning. (inconsistent synchronization)

        I did a quick look at your patch and I am a bit confused by the implementation of setSignaled. I realize that you did not really change the code at all, but is there a reason that the parameter is ignored? If you don't know I can look into it myself.

        Show
        revans2 Robert Joseph Evans added a comment - Can you please address the findbugs warning. (inconsistent synchronization) I did a quick look at your patch and I am a bit confused by the implementation of setSignaled. I realize that you did not really change the code at all, but is there a reason that the parameter is ignored? If you don't know I can look into it myself.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12520970/MAPREDUCE-4071.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
        org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520970/MAPREDUCE-4071.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2148//console This message is automatically generated.
        Hide
        revans2 Robert Joseph Evans added a comment -

        Putting in Patch Available so that Jenkins can test it.

        Show
        revans2 Robert Joseph Evans added a comment - Putting in Patch Available so that Jenkins can test it.
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        Attaching patch. Please review

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - Attaching patch. Please review
        Hide
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment -

        MRAppMaster received shutdown signal just before executing start method of ContainerAllocatorRouter service. So while executing shutdown hook
        of MRAppMaster

        MRAppMaster.java
        if(appMaster.containerAllocator instanceof ContainerAllocatorRouter) {
                ((ContainerAllocatorRouter) appMaster.containerAllocator)
                .setSignalled(true);
              }
        

        above code tries set the flag of containerAllocator object in ContainerAllocatorRouter#setSignalled(). But containerAllocator object has been instantiated while executing start method. So containerAllocator is null and hence NPE.

        Show
        kam_iitkgp Bhallamudi Venkata Siva Kamesh added a comment - MRAppMaster received shutdown signal just before executing start method of ContainerAllocatorRouter service. So while executing shutdown hook of MRAppMaster MRAppMaster.java if (appMaster.containerAllocator instanceof ContainerAllocatorRouter) { ((ContainerAllocatorRouter) appMaster.containerAllocator) .setSignalled( true ); } above code tries set the flag of containerAllocator object in ContainerAllocatorRouter#setSignalled(). But containerAllocator object has been instantiated while executing start method. So containerAllocator is null and hence NPE.

          People

          • Assignee:
            Unassigned
            Reporter:
            kam_iitkgp Bhallamudi Venkata Siva Kamesh
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development