[HBASE-15657] Failed snapshot verification may not be detected by TakeSnapshotHandler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 14/Apr/16 19:26
• Updated: 03/May/16 16:16
• Resolved: 03/May/16 16:16

Description

2016-04-13 07:41:09,572 INFO  [(10.0.0.75,16000,1460531913905)-proc-coordinator-pool1-thread-1] procedure.Procedure: Procedure 'snapshot_tb1_create' execution completed
2016-04-13 07:41:09,573 DEBUG [(10.0.0.75,16000,1460531913905)-proc-coordinator-pool1-thread-1] procedure.Procedure: Running finish phase.
2016-04-13 07:41:09,573 DEBUG [(10.0.0.75,16000,1460531913905)-proc-coordinator-pool1-thread-1] procedure.Procedure: Finished coordinator procedure - removing self from list of  running procedures
2016-04-13 07:41:09,573 DEBUG [(10.0.0.75,16000,1460531913905)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs: Attempting to clean out zk node for op:     snapshot_tb1_create
2016-04-13 07:41:09,573 INFO  [(10.0.0.75,16000,1460531913905)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil: Clearing all znodes for procedure

Encountered a case where snapshot verification failed:

2016-04-13 07:41:12,308 ERROR [MASTER_TABLE_OPERATIONS-10.0.0.75:16000-0] executor.EventHandler: Caught throwable while processing event C_M_SNAPSHOT_TABLE
java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos
  at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.internalGetFieldAccessorTable(SnapshotProtos.java:3883)
  at com.google.protobuf.GeneratedMessage.getDescriptorForType(GeneratedMessage.java:98)
  at com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.java:789)
  at com.google.protobuf.AbstractMessage$Builder.findMissingFields(AbstractMessage.java:780)
  at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
  at com.google.protobuf.AbstractMessage.newUninitializedMessageException(AbstractMessage.java:237)
  at com.google.protobuf.AbstractParser.newUninitializedMessageException(AbstractParser.java:57)
  at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
  at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
  at org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:433)
  at org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:273)
  at org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
  at org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:108)
  at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:200)
  at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)

Note NoClassDefFoundError is not Exception. So it was not caught by the following clause in TakeSnapshotHandler :

    } catch (Exception e) {