[HBASE-11878] TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController not yet initialized

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.99.0, 0.98.6
• Hadoop Flags: Reviewed

Release Note

A new exception, VisibilityControllerNotReadyException, is introduced for the case where VisibilityController is being initialized. Client can respond to this exception by retrying prior operation.
VisibilityControllerNotReadyException isn't a DoNotRetryIOException.

• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 02/Sep/14 14:33
• Updated: 06/Apr/18 17:51
• Resolved: 02/Sep/14 23:07

Description

In the thread w.r.t. first RC of 0.98.6,
http://search-hadoop.com/m/DHED4p2rw81 , Jean-Marc reported that TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes failed on his machines.

From http://server.distparser.com:81/hbase/with_teds_patch2/hbasetest1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt :

result {
  exception {
    name: "org.apache.hadoop.hbase.coprocessor.CoprocessorException"
    value: "org.apache.hadoop.hbase.coprocessor.CoprocessorException: VisibilityController not yet initialized\n\tat org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:638)\n\tat org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)\n\tat org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5794)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1608)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1590)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30088)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2014)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)\n\tat java.lang.Thread.run(Thread.java:744)\n"
  }
}

The above exception revealed a race condition: writing of labels "ABC" and "XYZ" took place when VisibilityController was not yet initialized.
The test writes the labels only once, leading to assertion failure.

Issue Links

• is related to: HBASE-11918 TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController initialization not being recognized