[HDFS-13065] TestErasureCodingMultipleRacks#testSkewedRack3 is failing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0, 3.0.1
• Component/s: hdfs
• Target Version/s: 3.0.0
• Hadoop Flags: Reviewed
• Assignee: Gabor Bota
• Reporter: Gabor Bota
• Due: 25/Jan/18
• Created: 25/Jan/18 10:28
• Updated: 29/Jan/18 06:31
• Resolved: 29/Jan/18 06:14

Description

Test fails intermittently.

java.lang.AssertionError: expected:<9> but was:<7>
at org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178)

2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG net.NetworkTopology (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node to choose.
2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) nodes evenly across racks, falling back to evenly place on the remaining racks. This may not guarantee rack-level fault tolerance. Please check if the racks are configured properly.
2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - Caught exception was:
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: 
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)