Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9458

Intermittent TestFlushSnapshotFromClient#testTakeSnapshotAfterMerge failure

    XMLWordPrintableJSON

Details

    • Test
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.0
    • None
    • None
    • Reviewed

    Description

      From https://builds.apache.org/job/HBase-0.96/20/testReport/org.apache.hadoop.hbase.snapshot/TestFlushSnapshotFromClient/testTakeSnapshotAfterMerge/ :

      org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=snapshotAfterMerge table=test type=FLUSH } had an error.  Procedure snapshotAfterMerge { waiting=[] done=[] }
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
      	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:210)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:221)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3120)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2672)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2605)
      	at org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:2612)
      	at org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient.testTakeSnapshotAfterMerge(TestFlushSnapshotFromClient.java:336)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
      Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=snapshotAfterMerge table=test type=FLUSH } had an error.  Procedure snapshotAfterMerge { waiting=[] done=[] }
      	at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:365)
      	at org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2947)
      	at org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$2.callBlockingMethod(MasterAdminProtos.java:32890)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
      	at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
      Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException via Failed taking snapshot { ss=snapshotAfterMerge table=test type=FLUSH } due to exception:Missing parent hfile for: 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e:org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Missing parent hfile for: 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
      	at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
      	at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:318)
      	at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:355)
      	... 4 more
      Caused by: org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Missing parent hfile for: 30b951996ef34885a2f5d64e4acb2467.6d1ed72bf95759cb606e8a6efdc6908e
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyStoreFile(MasterSnapshotVerifier.java:224)
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.access$000(MasterSnapshotVerifier.java:82)
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier$1.storeFile(MasterSnapshotVerifier.java:210)
      	at org.apache.hadoop.hbase.util.FSVisitor.visitRegionStoreFiles(FSVisitor.java:115)
      	at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitRegionStoreFiles(SnapshotReferenceUtil.java:123)
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegion(MasterSnapshotVerifier.java:207)
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:175)
      	at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:120)
      	at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:189)
      	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:131)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:724)
      
      	at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1425)
      	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1629)
      	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1686)
      	at org.apache.hadoop.hbase.protobuf.generated.MasterAdminProtos$MasterAdminService$BlockingStub.isSnapshotDone(MasterAdminProtos.java:34923)
      	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.isSnapshotDone(HConnectionManager.java:2123)
      	at org.apache.hadoop.hbase.client.HBaseAdmin$24.call(HBaseAdmin.java:2675)
      	at org.apache.hadoop.hbase.client.HBaseAdmin$24.call(HBaseAdmin.java:2672)
      	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:118)
      

      Attachments

        1. 9458-v1.txt
          3 kB
          Ted Yu
        2. 9458-v2.txt
          3 kB
          Ted Yu

        Activity

          People

            yuzhihong@gmail.com Ted Yu
            yuzhihong@gmail.com Ted Yu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: