Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11878

TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes fails due to VisibilityController not yet initialized

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.6
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A new exception, VisibilityControllerNotReadyException, is introduced for the case where VisibilityController is being initialized. Client can respond to this exception by retrying prior operation.
      VisibilityControllerNotReadyException isn't a DoNotRetryIOException.
      Show
      A new exception, VisibilityControllerNotReadyException, is introduced for the case where VisibilityController is being initialized. Client can respond to this exception by retrying prior operation. VisibilityControllerNotReadyException isn't a DoNotRetryIOException.

      Description

      In the thread w.r.t. first RC of 0.98.6,
      http://search-hadoop.com/m/DHED4p2rw81 , Jean-Marc reported that TestVisibilityLabelsWithDistributedLogReplay#testAddVisibilityLabelsOnRSRestart sometimes failed on his machines.

      From http://server.distparser.com:81/hbase/with_teds_patch2/hbasetest1/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay-output.txt :

      result {
        exception {
          name: "org.apache.hadoop.hbase.coprocessor.CoprocessorException"
          value: "org.apache.hadoop.hbase.coprocessor.CoprocessorException: VisibilityController not yet initialized\n\tat org.apache.hadoop.hbase.security.visibility.VisibilityController.addLabels(VisibilityController.java:638)\n\tat org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService$1.addLabels(VisibilityLabelsProtos.java:5014)\n\tat org.apache.hadoop.hbase.protobuf.generated.VisibilityLabelsProtos$VisibilityLabelsService.callMethod(VisibilityLabelsProtos.java:5178)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5794)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1608)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1590)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30088)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2014)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)\n\tat java.lang.Thread.run(Thread.java:744)\n"
        }
      }
      

      The above exception revealed a race condition: writing of labels "ABC" and "XYZ" took place when VisibilityController was not yet initialized.
      The test writes the labels only once, leading to assertion failure.

        Attachments

        1. 11878-v1.txt
          4 kB
          Ted Yu
        2. 11878-v2.txt
          8 kB
          Ted Yu
        3. 11878-v2.txt
          8 kB
          Ted Yu
        4. 11878-v3.txt
          7 kB
          Ted Yu
        5. 11878-v4.txt
          9 kB
          Ted Yu
        6. 11878-v5.txt
          9 kB
          Ted Yu

          Issue Links

            Activity

              People

              • Assignee:
                yuzhihong@gmail.com Ted Yu
                Reporter:
                yuzhihong@gmail.com Ted Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: