Ravi, thanks for taking this up.

How does it relate to MAPREDUCE-2956?

Hi Arun,
I think this issue doesn't hold any relation with MAPREDUCE-2956. As MAPREDUCE-2956 speaks of failure while stopping a service , but this issue deals with the startup failure of a service.

All the composite services are initialized (not started) at the time of NodeManager initialization. As part of this, the ResourceLocalizationService is initialized , which starts PublicLocalizer Thread. This thread waits for the CompletionService queue.

After the services are initialized,while services startup, NodeStatusUpdaterImpl service startup failed, so only the Deletion Service (which is started before it) is stopped. Hence the PublicLocalizer thread is still running, which is runs the NodeManager even though no service is started.

"Thread-11" prio=10 tid=0x70468800 nid=0x11ce waiting on condition [0x706fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0xa01f4a40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:164)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.run(ResourceLocalizationService.java:547)
    

As part of this, the ResourceLocalizationService is initialized , which starts PublicLocalizer Thread.

This is the bug that should be fixed. The service lifecycle clearly intends init() to only initialize services. Threads should be started only via start() so that stop() can properly handle their exit. We should just make ResourceLocalizationService.LocalizerTracker itself a service and follow the above life-cycle pattern.

Can you look at your thread dump and name the other threads that are hanging like this? Thanks!

Attached the thread dump taken after the services are stopped.

We have two problems with the ResourceLocalizationService initialization
1.Starting of the PublicLocalizer thread.
2.CacheCleanup timer is scheduled.

These two threads are blocking the JVM exit.
Ideally these are meant to be done when the service is started.

Hi Vinod,
Can we perform these tasks at the startup of the service rather than at initialization.

I think LocalizerTracker should be under the ResourceLocalizationService as its not generic enough, to be made a seperate service.I feel that moving the tasks to the service startup may be a good thing.

Thanks.

Can we perform these tasks at the startup of the service rather than at initialization.

Yes, sure. Can you provide a patch, thanks!

Attached a patch.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493995/MAPREDUCE-2949.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/700//console

This message is automatically generated.

Have manually verified the shutdown scenario of the Nodemanager when the service startup failed after applying the patch.

Thanks for the patch Ravi! Couple of comments:

I think LocalizerTracker should be under the ResourceLocalizationService as its not generic enough, to be made a seperate service.I feel that moving the tasks to the service startup may be a good thing.

What I meant was that LocalizerTracker can be under ResourceLocalizationService but still extends AbstractService and thus implement the life-cycle properly, that's been the convention and discipline we've been adhering to in YARN

Granted the localizerTracker is going away because of NM's shutdown hook, one thing that is not clear is how the cacheCleanup executor-service is going away during JVM shutdown in your manual verification after the patch. May be it is the timing and NM will probably shut down cleanly if there are no active thread in the pool? I'd think that we should do an explicit shutdown on the executor-service.

Hi Vinod, Thanks for correcting me, I have got it wrong the first time.

Sorry for the delay,Attached the patch with the review comments fixed.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12494445/MAPREDUCE-2949.1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/735//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/735//console

This message is automatically generated.

Looks good, +1.

I just committed this to trunk and branch 0.23. Thanks Ravi Teja!

Integrated in Hadoop-Mapreduce-trunk-Commit #899 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/899/)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V.

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Common-trunk-Commit #887 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/887/)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V.

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Hdfs-trunk-Commit #964 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/964/)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V.

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Mapreduce-0.23-Build #2 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/2/)
MAPREDUCE-2949. svn merge -c r1170689 --ignore-ancestry ../../trunk/

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170690
Files :

• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Hdfs-0.23-Build #7 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/7/)
MAPREDUCE-2949. svn merge -c r1170689 --ignore-ancestry ../../trunk/

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170690
Files :

• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Mapreduce-trunk #831 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/831/)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V.

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java

Integrated in Hadoop-Hdfs-trunk #801 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/801/)
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service startup fails. Contributed by Ravi Teja Ch N V.

vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170689
Files :

• /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java