Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11918

TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController initialization not being recognized

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.7
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Here is one example:
      https://builds.apache.org/job/hbase-0.98/lastCompletedBuild/testReport/org.apache.hadoop.hbase.security.visibility/TestVisibilityLabelsWithDistributedLogReplay/testAddVisibilityLabelsOnRSRestart/

      2014-09-09 02:46:05,168 DEBUG [Thread-245] visibility.TestVisibilityLabelsWithDefaultVisLabelService$2(127): Got exception writing labels
      org.apache.hadoop.hbase.security.visibility.VisibilityControllerNotReadyException: org.apache.hadoop.hbase.security.visibility.VisibilityControllerNotReadyException: VisibilityController not yet initialized!
      	at org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:644)
      	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)
      	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)
      	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5591)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3396)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3378)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29591)
      ...
      2014-09-09 02:46:10,087 DEBUG [Thread-245] visibility.TestVisibilityLabelsWithDefaultVisLabelService$2(127): Got exception writing labels
      org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.security.visibility.LabelAlreadyExistsException: Label 'secret' already exists
      	at org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:667)
      	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)
      	at org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)
      	at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5591)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3396)
      	at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3378)
      	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29591)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
      	at java.lang.Thread.run(Thread.java:662)
      
      	at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1460)
      	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService$2.run(TestVisibilityLabelsWithDefaultVisLabelService.java:126)
      	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService$2.run(TestVisibilityLabelsWithDefaultVisLabelService.java:118)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
      	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:268)
      	at org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService.testAddVisibilityLabelsOnRSRestart(TestVisibilityLabelsWithDefaultVisLabelService.java:140)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
      

      Here is related code:

                    if (t instanceof VisibilityControllerNotReadyException) {
                      vcInitialized.set(false);
                      LOG.warn("VisibilityController was not yet initialized");
                      Threads.sleep(10);
                    }
      

      vcInitialized needs to be set true when VisibilityController initialization is done.

        Attachments

        1. 11918-v1.txt
          1.0 kB
          Ted Yu
        2. 11918-v1.txt
          1.0 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: