[HBASE-7344] subprocedure initialization fails with invalid znode data.

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Jonathan Hsieh
• Created: 13/Dec/12 15:27
• Updated: 15/Jun/22 20:51
• Resolved: 09/Jan/13 07:49

Description

Sometimes snapshots subprocedures fail to start on RS because data read from ZK is bad.

2012-12-13 07:22:55,238 ERROR org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs: Illegal argument exception
java.lang.IllegalArgumentException: Could not read snapshot information from request.
        at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedureBuilder.buildSubprocedure(RegionServerSnapsh
otManager.java:284)
        at org.apache.hadoop.hbase.procedure.ProcedureMember.createSubprocedure(ProcedureMember.java:98)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:199)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.waitForNewProcedures(ZKProcedureMemberRpcs.java:167)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$1(ZKProcedureMemberRpcs.java:150)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:106)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
2012-12-13 07:22:55,239 ERROR org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs: Failed due to null subprocedure
Local ForeignThreadException from null
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:203)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.waitForNewProcedures(ZKProcedureMemberRpcs.java:167)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$1(ZKProcedureMemberRpcs.java:150)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:106)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
Caused by: java.lang.IllegalArgumentException: Could not read snapshot information from request.
        at org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedureBuilder.buildSubprocedure(RegionServerSnapshotManager.java:284)
        at org.apache.hadoop.hbase.procedure.ProcedureMember.createSubprocedure(ProcedureMember.java:98)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:199)
        ... 6 more
2012-12-13 07:22:55,239 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher 
java.lang.NullPointerException
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberAborted(ZKProcedureMemberRpcs.java:266)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:203)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.waitForNewProcedures(ZKProcedureMemberRpcs.java:167)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$1(ZKProcedureMemberRpcs.java:150)
        at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:106)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:303)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)