Hadoop Common
  1. Hadoop Common
  2. HADOOP-10227

IPC shutdown hangs if remote connection not reachable

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: ipc
    • Labels:
      None
    • Environment:

      OS/X Java7

      Description

      If the remote HDFS namenode isn't reachable, and an attempt to shut down the JVM is made, the process will keep running

        Activity

        Steve Loughran created issue -
        Hide
        Steve Loughran added a comment -
        
        "Thread-0" prio=5 tid=0x00007fb05a077000 nid=0x5d0f waiting on condition [0x0000000116565000]
           java.lang.Thread.State: TIMED_WAITING (sleeping)
        	at java.lang.Thread.sleep(Native Method)
        	at org.apache.hadoop.ipc.Client.stop(Client.java:1173)
        	at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
        	at org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
        	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.close(ClientNamenodeProtocolTranslatorPB.java:174)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621)
        	at org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:738)
        	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:794)
        	- locked <0x00000007fec77980> (a org.apache.hadoop.hdfs.DFSClient)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:847)
        	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524)
        	- locked <0x00000007fec76600> (a org.apache.hadoop.fs.FileSystem$Cache)
        	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541)
        	- locked <0x00000007fec76618> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer)
        	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
        
        "SIGINT handler" daemon prio=5 tid=0x00007fb0588f7000 nid=0x440f in Object.wait() [0x00000001138bb000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1280)
        	- locked <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1354)
        	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
        	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
        	at java.lang.Shutdown.runHooks(Shutdown.java:123)
        	at java.lang.Shutdown.sequence(Shutdown.java:167)
        	at java.lang.Shutdown.exit(Shutdown.java:212)
        	- locked <0x00000007fed5dfe8> (a java.lang.Class for java.lang.Shutdown)
        	at java.lang.Runtime.exit(Runtime.java:109)
        	at java.lang.System.exit(System.java:962)
        	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:133)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.exit(ServiceLauncher.java:279)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.interrupted(ServiceLauncher.java:266)
        	at org.apache.hadoop.yarn.service.launcher.IrqHandler.handle(IrqHandler.java:70)
        	at sun.misc.Signal$1.run(Signal.java:212)
        	at java.lang.Thread.run(Thread.java:744)
        
        "Service Thread" daemon prio=5 tid=0x00007fb059011000 nid=0x5303 runnable [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "C2 CompilerThread1" daemon prio=5 tid=0x00007fb05b002000 nid=0x5103 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "C2 CompilerThread0" daemon prio=5 tid=0x00007fb05b000000 nid=0x4f03 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "Signal Dispatcher" daemon prio=5 tid=0x00007fb058826800 nid=0x4d03 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "Finalizer" daemon prio=5 tid=0x00007fb05a04e000 nid=0x3903 in Object.wait() [0x0000000113772000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        	- locked <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
        
        "Reference Handler" daemon prio=5 tid=0x00007fb05a04b000 nid=0x3703 in Object.wait() [0x000000011366f000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007feb104b8> (a java.lang.ref.Reference$Lock)
        	at java.lang.Object.wait(Object.java:503)
        	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        	- locked <0x00000007feb104b8> (a java.lang.ref.Reference$Lock)
        
        "main" prio=5 tid=0x00007fb059002800 nid=0x1903 runnable [0x000000010b545000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
        	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        	- locked <0x00000007fb6a4260> (a sun.nio.ch.Util$2)
        	- locked <0x00000007fb6a41e0> (a java.util.Collections$UnmodifiableSet)
        	- locked <0x00000007fb6a4058> (a sun.nio.ch.KQueueSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
        	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
        	- locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection)
        	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
        	- locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection)
        	at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
        	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1318)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        	at com.sun.proxy.$Proxy13.getApplications(Unknown Source)
        	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:197)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        	at com.sun.proxy.$Proxy14.getApplications(Unknown Source)
        	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:237)
        	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:221)
        	at org.apache.hadoop.hoya.yarn.client.HoyaYarnClientImpl.listHoyaInstances(HoyaYarnClientImpl.java:75)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.listHoyaInstances(HoyaClient.java:1167)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.actionList(HoyaClient.java:1179)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.runService(HoyaClient.java:234)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:178)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:394)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:323)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.serviceMain(ServiceLauncher.java:539)
        	at org.apache.hadoop.hoya.Hoya.main(Hoya.java:49)
        
        Show
        Steve Loughran added a comment - " Thread -0" prio=5 tid=0x00007fb05a077000 nid=0x5d0f waiting on condition [0x0000000116565000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.ipc.Client.stop(Client.java:1173) at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626) at org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57) at org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.close(ClientNamenodeProtocolTranslatorPB.java:174) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621) at org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:738) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:794) - locked <0x00000007fec77980> (a org.apache.hadoop.hdfs.DFSClient) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:847) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524) - locked <0x00000007fec76600> (a org.apache.hadoop.fs.FileSystem$Cache) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541) - locked <0x00000007fec76618> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) "SIGINT handler" daemon prio=5 tid=0x00007fb0588f7000 nid=0x440f in Object .wait() [0x00000001138bb000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1280) - locked <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1354) at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106) at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) at java.lang.Shutdown.runHooks(Shutdown.java:123) at java.lang.Shutdown.sequence(Shutdown.java:167) at java.lang.Shutdown.exit(Shutdown.java:212) - locked <0x00000007fed5dfe8> (a java.lang. Class for java.lang.Shutdown) at java.lang. Runtime .exit( Runtime .java:109) at java.lang. System .exit( System .java:962) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:133) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.exit(ServiceLauncher.java:279) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.interrupted(ServiceLauncher.java:266) at org.apache.hadoop.yarn.service.launcher.IrqHandler.handle(IrqHandler.java:70) at sun.misc.Signal$1.run(Signal.java:212) at java.lang. Thread .run( Thread .java:744) "Service Thread " daemon prio=5 tid=0x00007fb059011000 nid=0x5303 runnable [0x0000000000000000] java.lang. Thread .State: RUNNABLE "C2 CompilerThread1" daemon prio=5 tid=0x00007fb05b002000 nid=0x5103 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "C2 CompilerThread0" daemon prio=5 tid=0x00007fb05b000000 nid=0x4f03 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "Signal Dispatcher" daemon prio=5 tid=0x00007fb058826800 nid=0x4d03 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "Finalizer" daemon prio=5 tid=0x00007fb05a04e000 nid=0x3903 in Object .wait() [0x0000000113772000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) "Reference Handler" daemon prio=5 tid=0x00007fb05a04b000 nid=0x3703 in Object .wait() [0x000000011366f000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007feb104b8> (a java.lang.ref.Reference$Lock) at java.lang. Object .wait( Object .java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000007feb104b8> (a java.lang.ref.Reference$Lock) "main" prio=5 tid=0x00007fb059002800 nid=0x1903 runnable [0x000000010b545000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000007fb6a4260> (a sun.nio.ch.Util$2) - locked <0x00000007fb6a41e0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007fb6a4058> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547) - locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642) - locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection) at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1318) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy13.getApplications(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy14.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:237) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:221) at org.apache.hadoop.hoya.yarn.client.HoyaYarnClientImpl.listHoyaInstances(HoyaYarnClientImpl.java:75) at org.apache.hadoop.hoya.yarn.client.HoyaClient.listHoyaInstances(HoyaClient.java:1167) at org.apache.hadoop.hoya.yarn.client.HoyaClient.actionList(HoyaClient.java:1179) at org.apache.hadoop.hoya.yarn.client.HoyaClient.runService(HoyaClient.java:234) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:178) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:394) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:323) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.serviceMain(ServiceLauncher.java:539) at org.apache.hadoop.hoya.Hoya.main(Hoya.java:49)
        Hide
        Steve Loughran added a comment -

        resolving as duplicate of HADOOP-10219. Sorry -I needed to rerun the video I was making on filing issues and this was the bug I had to hand

        Show
        Steve Loughran added a comment - resolving as duplicate of HADOOP-10219 . Sorry -I needed to rerun the video I was making on filing issues and this was the bug I had to hand
        Steve Loughran made changes -
        Field Original Value New Value
        Resolution Duplicate [ 3 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        6m 38s 1 Steve Loughran 12/Jan/14 21:26

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development