Hadoop Common
  1. Hadoop Common
  2. HADOOP-8063

Hadoop balancer will creart too many threds will crash the balancer JVM

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:

      Description

      The balancer process will create too many thread will more than the jvm and the jvm will crash.

        Activity

        Hide
        sutao bian added a comment -

        2012-02-10 13:13:50
        Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode):

        "pool-1-thread-689" prio=10 tid=0x00007f7fd4029000 nid=0x62e2 runnable [0x00007f81cadec000..0x00007f81cadeca90]
        java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)

        • locked <0x00007f8283d10778> (a java.io.BufferedInputStream)
          at java.io.DataInputStream.readShort(DataInputStream.java:295)
          at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.receiveResponse(Balancer.java:367)
          at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.dispatch(Balancer.java:322)
          at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.access$1800(Balancer.java:228)
          at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove$1.run(Balancer.java:389)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:

        • <0x00007f8283d036b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

        "pool-1-thread-688" prio=10 tid=0x00007f7fd4027c00 nid=0x62e1 waiting on condition [0x00007f81caeed000..0x00007f81caeed810]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)

        • parking to wait for <0x00007f82300ec758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
          at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:

        • None

        "pool-1-thread-687" prio=10 tid=0x00007f7fd4026400 nid=0x62e0 waiting on condition [0x00007f81cafee000..0x00007f81cafee790]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)

        • parking to wait for <0x00007f82300ec758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
          at java.lang.Thread.run(Thread.java:619)

        Locked ownable synchronizers:

        • None
        Show
        sutao bian added a comment - 2012-02-10 13:13:50 Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode): "pool-1-thread-689" prio=10 tid=0x00007f7fd4029000 nid=0x62e2 runnable [0x00007f81cadec000..0x00007f81cadeca90] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) locked <0x00007f8283d10778> (a java.io.BufferedInputStream) at java.io.DataInputStream.readShort(DataInputStream.java:295) at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.receiveResponse(Balancer.java:367) at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.dispatch(Balancer.java:322) at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove.access$1800(Balancer.java:228) at org.apache.hadoop.hdfs.server.balancer.Balancer$PendingBlockMove$1.run(Balancer.java:389) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: <0x00007f8283d036b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "pool-1-thread-688" prio=10 tid=0x00007f7fd4027c00 nid=0x62e1 waiting on condition [0x00007f81caeed000..0x00007f81caeed810] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) parking to wait for <0x00007f82300ec758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: None "pool-1-thread-687" prio=10 tid=0x00007f7fd4026400 nid=0x62e0 waiting on condition [0x00007f81cafee000..0x00007f81cafee790] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) parking to wait for <0x00007f82300ec758> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: None
        Hide
        sutao bian added a comment -

        Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
        Feb 10, 2012 10:41:13 AM 0 0 KB 115.57 GB 30 GB
        Feb 10, 2012 11:03:44 AM 1 3.66 GB 111.77 GB 30 GB
        Feb 10, 2012 11:26:44 AM 2 7.25 GB 108.21 GB 30 GB
        Feb 10, 2012 11:49:45 AM 3 10.92 GB 104.51 GB 30 GB
        Feb 10, 2012 12:11:45 PM 4 14.23 GB 101.18 GB 30 GB
        Feb 10, 2012 12:33:46 PM 5 17.85 GB 97.7 GB 30 GB
        Feb 10, 2012 12:55:16 PM 6 21.48 GB 94.43 GB 30 GB
        Feb 10, 2012 1:17:17 PM 7 25.1 GB 90.45 GB 30 GB

        Show
        sutao bian added a comment - Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved Feb 10, 2012 10:41:13 AM 0 0 KB 115.57 GB 30 GB Feb 10, 2012 11:03:44 AM 1 3.66 GB 111.77 GB 30 GB Feb 10, 2012 11:26:44 AM 2 7.25 GB 108.21 GB 30 GB Feb 10, 2012 11:49:45 AM 3 10.92 GB 104.51 GB 30 GB Feb 10, 2012 12:11:45 PM 4 14.23 GB 101.18 GB 30 GB Feb 10, 2012 12:33:46 PM 5 17.85 GB 97.7 GB 30 GB Feb 10, 2012 12:55:16 PM 6 21.48 GB 94.43 GB 30 GB Feb 10, 2012 1:17:17 PM 7 25.1 GB 90.45 GB 30 GB

          People

          • Assignee:
            Unassigned
            Reporter:
            sutao bian
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development