[HBASE-11918] TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController initialization not being recognized

• Type: Test
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.99.0, 0.98.7
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 09/Sep/14 04:09
• Updated: 06/Apr/18 17:50
• Resolved: 09/Sep/14 14:35

Description

Here is one example:
https://builds.apache.org/job/hbase-0.98/lastCompletedBuild/testReport/org.apache.hadoop.hbase.security.visibility/TestVisibilityLabelsWithDistributedLogReplay/testAddVisibilityLabelsOnRSRestart/

2014-09-09 02:46:05,168 DEBUG [Thread-245] visibility.TestVisibilityLabelsWithDefaultVisLabelService$2(127): Got exception writing labels
org.apache.hadoop.hbase.security.visibility.VisibilityControllerNotReadyException: org.apache.hadoop.hbase.security.visibility.VisibilityControllerNotReadyException: VisibilityController not yet initialized!
	at org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:644)
	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)
	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)
	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5591)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3396)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3378)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29591)
...
2014-09-09 02:46:10,087 DEBUG [Thread-245] visibility.TestVisibilityLabelsWithDefaultVisLabelService$2(127): Got exception writing labels
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.security.visibility.LabelAlreadyExistsException: Label 'secret' already exists
	at org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:667)
	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)
	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)
	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5591)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3396)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3378)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29591)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
	at java.lang.Thread.run(Thread.java:662)

	at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1460)
	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService$2.run(TestVisibilityLabelsWithDefaultVisLabelService.java:126)
	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService$2.run(TestVisibilityLabelsWithDefaultVisLabelService.java:118)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:268)
	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabelsWithDefaultVisLabelService.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Here is related code:

              if (t instanceof VisibilityControllerNotReadyException) {
                vcInitialized.set(false);
                LOG.warn("VisibilityController was not yet initialized");
                Threads.sleep(10);
              }

vcInitialized needs to be set true when VisibilityController initialization is done.

Issue Links

• relates to: HBASE-11878 TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController not yet initialized