[HDFS-10336] TestBalancer failing intermittently because of not reseting UserGroupInformation completely

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Yiqun Lin
• Reporter: Yiqun Lin
• Created: 27/Apr/16 03:53
• Updated: 06/Jan/17 08:03
• Resolved: 08/Jul/16 21:42

Description

The unit test TestBalancer failed sometimes.

I looked for the reason. I found two main reasons causing this.

• 1st. The test TestBalancer#testBalancerWithKeytabs executed timeout.

  org.apache.hadoop.hdfs.server.balancer.TestBalancer
  testBalancerWithKeytabs(org.apache.hadoop.hdfs.server.balancer.TestBalancer)  Time elapsed: 300.41 sec  <<< ERROR!
  java.lang.Exception: test timed out after 300000 milliseconds
  	at java.lang.Thread.sleep(Native Method)
  	at org.apache.hadoop.hdfs.server.balancer.Dispatcher.waitForMoveCompletion(Dispatcher.java:1122)
  	at org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchBlockMoves(Dispatcher.java:1096)
  	at org.apache.hadoop.hdfs.server.balancer.Dispatcher.dispatchAndCheckContinue(Dispatcher.java:1060)
  	at org.apache.hadoop.hdfs.server.balancer.Balancer.runOneIteration(Balancer.java:635)
  	at org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:689)
  	at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:1098)
  	at org.apache.hadoop.hdfs.server.balancer.TestBalancer.access$000(TestBalancer.java:125)
  

• 2nd. The test TestBalancer#testBalancerWithKeytabs reset the UGI not completely sometimes in the finally block. And this affected the other unit tests threw IOException, like this:

  testBalancerWithNonZeroThreadsForMove(org.apache.hadoop.hdfs.server.balancer.TestBalancer)  Time elapsed: 0 sec  <<< ERROR!
  java.io.IOException: Running in secure mode, but config doesn't have a keytab
  	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:300)
  

  And there were not only one test will be affected by this. We should add a line to do before doing reset UGI operation and can avoid the potenial exception happens.

  UserGroupInformation.reset();
  

Issue Links

• is related to: HDFS-10602 TestBalancer runs timeout intermittently
• relates to: HDFS-9804 Allow long-running Balancer to login with keytab