Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3052

HA: hdfs balancer throws StandbyException


    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: ha
    • Labels:
    • Target Version/s:


      The hdfs balancer tool throws a StandbyException.

      Originally, styx01 hosts the active NN and styx02 hosts the standby NN.

      After failing over from styx01 NN to styx02 NN, the hdfs balancer command thows a StandbyException:

      12/03/06 00:34:01 INFO balancer.Balancer: namenodes = {ha-nn-uri={nn1=styx01.sf.cloudera.com/, nn2=styx02.sf.cloudera.com/}}
      12/03/06 00:34:01 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
      Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
      org.apache.hadoop.ipc.StandbyException: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
      	at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1028)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:653)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1522)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:437)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:254)
      	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42590)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:448)
      	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1622)
      	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1618)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
      	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1616)
      .  Exiting ...
      Balancing took 650.0 milliseconds

      After failing back so that active is on styx01 and standby is on styx02, the hdfs balancer command runs without exception.

      Failing over again results in the same StandbyException.

      Service ID nn1 corresponds to node styx01, and nn2 corresponds to styx02. Console output from styx01 and styx02 is attached.

      1. balancer_styx02
        6 kB
        Stephen Chu
      2. balancer_styx01
        6 kB
        Stephen Chu


        Stephen Chu created issue -
        Stephen Chu made changes -
        Field Original Value New Value
        Attachment balancer_styx01 [ 12517208 ]
        Attachment balancer_styx02 [ 12517209 ]
        Stephen Chu made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Duplicate [ 3 ]
        Allen Wittenauer made changes -
        Affects Version/s 2.0.0-alpha [ 12320353 ]
        Affects Version/s 0.24.0 [ 12317653 ]


          • Assignee:
            Stephen Chu
          • Votes:
            0 Vote for this issue
            5 Start watching this issue


            • Created: