Hadoop Common
  1. Hadoop Common
  2. HADOOP-10227

IPC shutdown hangs if remote connection not reachable

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: ipc
    • Labels:
      None
    • Environment:

      OS/X Java7

      Description

      If the remote HDFS namenode isn't reachable, and an attempt to shut down the JVM is made, the process will keep running

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        6m 38s 1 Steve Loughran 12/Jan/14 21:26
        Steve Loughran made changes -
        Field Original Value New Value
        Resolution Duplicate [ 3 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Hide
        Steve Loughran added a comment -

        resolving as duplicate of HADOOP-10219. Sorry -I needed to rerun the video I was making on filing issues and this was the bug I had to hand

        Show
        Steve Loughran added a comment - resolving as duplicate of HADOOP-10219 . Sorry -I needed to rerun the video I was making on filing issues and this was the bug I had to hand
        Hide
        Steve Loughran added a comment -
        
        "Thread-0" prio=5 tid=0x00007fb05a077000 nid=0x5d0f waiting on condition [0x0000000116565000]
           java.lang.Thread.State: TIMED_WAITING (sleeping)
        	at java.lang.Thread.sleep(Native Method)
        	at org.apache.hadoop.ipc.Client.stop(Client.java:1173)
        	at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
        	at org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626)
        	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.close(ClientNamenodeProtocolTranslatorPB.java:174)
        	at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621)
        	at org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:738)
        	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:794)
        	- locked <0x00000007fec77980> (a org.apache.hadoop.hdfs.DFSClient)
        	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:847)
        	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524)
        	- locked <0x00000007fec76600> (a org.apache.hadoop.fs.FileSystem$Cache)
        	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541)
        	- locked <0x00000007fec76618> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer)
        	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
        
        "SIGINT handler" daemon prio=5 tid=0x00007fb0588f7000 nid=0x440f in Object.wait() [0x00000001138bb000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1280)
        	- locked <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1)
        	at java.lang.Thread.join(Thread.java:1354)
        	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
        	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
        	at java.lang.Shutdown.runHooks(Shutdown.java:123)
        	at java.lang.Shutdown.sequence(Shutdown.java:167)
        	at java.lang.Shutdown.exit(Shutdown.java:212)
        	- locked <0x00000007fed5dfe8> (a java.lang.Class for java.lang.Shutdown)
        	at java.lang.Runtime.exit(Runtime.java:109)
        	at java.lang.System.exit(System.java:962)
        	at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:133)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.exit(ServiceLauncher.java:279)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.interrupted(ServiceLauncher.java:266)
        	at org.apache.hadoop.yarn.service.launcher.IrqHandler.handle(IrqHandler.java:70)
        	at sun.misc.Signal$1.run(Signal.java:212)
        	at java.lang.Thread.run(Thread.java:744)
        
        "Service Thread" daemon prio=5 tid=0x00007fb059011000 nid=0x5303 runnable [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "C2 CompilerThread1" daemon prio=5 tid=0x00007fb05b002000 nid=0x5103 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "C2 CompilerThread0" daemon prio=5 tid=0x00007fb05b000000 nid=0x4f03 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "Signal Dispatcher" daemon prio=5 tid=0x00007fb058826800 nid=0x4d03 waiting on condition [0x0000000000000000]
           java.lang.Thread.State: RUNNABLE
        
        "Finalizer" daemon prio=5 tid=0x00007fb05a04e000 nid=0x3903 in Object.wait() [0x0000000113772000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        	- locked <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock)
        	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
        
        "Reference Handler" daemon prio=5 tid=0x00007fb05a04b000 nid=0x3703 in Object.wait() [0x000000011366f000]
           java.lang.Thread.State: WAITING (on object monitor)
        	at java.lang.Object.wait(Native Method)
        	- waiting on <0x00000007feb104b8> (a java.lang.ref.Reference$Lock)
        	at java.lang.Object.wait(Object.java:503)
        	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        	- locked <0x00000007feb104b8> (a java.lang.ref.Reference$Lock)
        
        "main" prio=5 tid=0x00007fb059002800 nid=0x1903 runnable [0x000000010b545000]
           java.lang.Thread.State: RUNNABLE
        	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
        	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
        	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        	- locked <0x00000007fb6a4260> (a sun.nio.ch.Util$2)
        	- locked <0x00000007fb6a41e0> (a java.util.Collections$UnmodifiableSet)
        	- locked <0x00000007fb6a4058> (a sun.nio.ch.KQueueSelectorImpl)
        	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        	at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
        	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
        	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
        	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
        	- locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection)
        	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
        	- locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection)
        	at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
        	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1318)
        	at org.apache.hadoop.ipc.Client.call(Client.java:1300)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        	at com.sun.proxy.$Proxy13.getApplications(Unknown Source)
        	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:197)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:606)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        	at com.sun.proxy.$Proxy14.getApplications(Unknown Source)
        	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:237)
        	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:221)
        	at org.apache.hadoop.hoya.yarn.client.HoyaYarnClientImpl.listHoyaInstances(HoyaYarnClientImpl.java:75)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.listHoyaInstances(HoyaClient.java:1167)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.actionList(HoyaClient.java:1179)
        	at org.apache.hadoop.hoya.yarn.client.HoyaClient.runService(HoyaClient.java:234)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:178)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:394)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:323)
        	at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.serviceMain(ServiceLauncher.java:539)
        	at org.apache.hadoop.hoya.Hoya.main(Hoya.java:49)
        
        Show
        Steve Loughran added a comment - " Thread -0" prio=5 tid=0x00007fb05a077000 nid=0x5d0f waiting on condition [0x0000000116565000] java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at org.apache.hadoop.ipc.Client.stop(Client.java:1173) at org.apache.hadoop.ipc.ClientCache.stopClient(ClientCache.java:100) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.close(ProtobufRpcEngine.java:251) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626) at org.apache.hadoop.io.retry.DefaultFailoverProxyProvider.close(DefaultFailoverProxyProvider.java:57) at org.apache.hadoop.io.retry.RetryInvocationHandler.close(RetryInvocationHandler.java:206) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:626) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.close(ClientNamenodeProtocolTranslatorPB.java:174) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:621) at org.apache.hadoop.hdfs.DFSClient.closeConnectionToNamenode(DFSClient.java:738) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:794) - locked <0x00000007fec77980> (a org.apache.hadoop.hdfs.DFSClient) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:847) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524) - locked <0x00000007fec76600> (a org.apache.hadoop.fs.FileSystem$Cache) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541) - locked <0x00000007fec76618> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) "SIGINT handler" daemon prio=5 tid=0x00007fb0588f7000 nid=0x440f in Object .wait() [0x00000001138bb000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1280) - locked <0x00000007fed3d2c8> (a org.apache.hadoop.util.ShutdownHookManager$1) at java.lang. Thread .join( Thread .java:1354) at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106) at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) at java.lang.Shutdown.runHooks(Shutdown.java:123) at java.lang.Shutdown.sequence(Shutdown.java:167) at java.lang.Shutdown.exit(Shutdown.java:212) - locked <0x00000007fed5dfe8> (a java.lang. Class for java.lang.Shutdown) at java.lang. Runtime .exit( Runtime .java:109) at java.lang. System .exit( System .java:962) at org.apache.hadoop.util.ExitUtil.terminate(ExitUtil.java:133) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.exit(ServiceLauncher.java:279) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.interrupted(ServiceLauncher.java:266) at org.apache.hadoop.yarn.service.launcher.IrqHandler.handle(IrqHandler.java:70) at sun.misc.Signal$1.run(Signal.java:212) at java.lang. Thread .run( Thread .java:744) "Service Thread " daemon prio=5 tid=0x00007fb059011000 nid=0x5303 runnable [0x0000000000000000] java.lang. Thread .State: RUNNABLE "C2 CompilerThread1" daemon prio=5 tid=0x00007fb05b002000 nid=0x5103 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "C2 CompilerThread0" daemon prio=5 tid=0x00007fb05b000000 nid=0x4f03 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "Signal Dispatcher" daemon prio=5 tid=0x00007fb058826800 nid=0x4d03 waiting on condition [0x0000000000000000] java.lang. Thread .State: RUNNABLE "Finalizer" daemon prio=5 tid=0x00007fb05a04e000 nid=0x3903 in Object .wait() [0x0000000113772000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000007feb00658> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) "Reference Handler" daemon prio=5 tid=0x00007fb05a04b000 nid=0x3703 in Object .wait() [0x000000011366f000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) - waiting on <0x00000007feb104b8> (a java.lang.ref.Reference$Lock) at java.lang. Object .wait( Object .java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000007feb104b8> (a java.lang.ref.Reference$Lock) "main" prio=5 tid=0x00007fb059002800 nid=0x1903 runnable [0x000000010b545000] java.lang. Thread .State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000007fb6a4260> (a sun.nio.ch.Util$2) - locked <0x00000007fb6a41e0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007fb6a4058> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547) - locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642) - locked <0x00000007fb5c4708> (a org.apache.hadoop.ipc.Client$Connection) at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1318) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy13.getApplications(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy14.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:237) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:221) at org.apache.hadoop.hoya.yarn.client.HoyaYarnClientImpl.listHoyaInstances(HoyaYarnClientImpl.java:75) at org.apache.hadoop.hoya.yarn.client.HoyaClient.listHoyaInstances(HoyaClient.java:1167) at org.apache.hadoop.hoya.yarn.client.HoyaClient.actionList(HoyaClient.java:1179) at org.apache.hadoop.hoya.yarn.client.HoyaClient.runService(HoyaClient.java:234) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:178) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceRobustly(ServiceLauncher.java:394) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.launchServiceAndExit(ServiceLauncher.java:323) at org.apache.hadoop.yarn.service.launcher.ServiceLauncher.serviceMain(ServiceLauncher.java:539) at org.apache.hadoop.hoya.Hoya.main(Hoya.java:49)
        Steve Loughran created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development