[HBASE-14783] Proc-V2: Master aborts when downgrading from 1.3 to 1.1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Later
• Assignee: Stephen Yuan Jiang
• Reporter: Ted Yu
• Created: 06/Nov/15 22:17
• Updated: 28/Aug/18 22:56
• Resolved: 28/Aug/18 22:56

Description

I was running ITBLL with 1.3 deployed on a 6 node cluster.

Then I stopped the cluster, deployed 1.1 release and tried to start cluster.
However, master failed to start due to:

2015-11-06 00:58:40,351 FATAL [eval-test-2:20000.activeMasterManager] master.HMaster: Failed to become active master
java.io.IOException: The procedure class org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure must be accessible and have an empty constructor
  at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:548)
  at org.apache.hadoop.hbase.procedure2.Procedure.convert(Procedure.java:640)
  at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormatReader.read(ProcedureWALFormatReader.java:105)
  at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.load(ProcedureWALFormat.java:82)
  at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.load(WALProcedureStore.java:298)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:275)
  at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:434)
  at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1208)
  at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1107)
  at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:694)
  at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:186)
  at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1713)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:191)
  at org.apache.hadoop.hbase.procedure2.Procedure.newInstance(Procedure.java:536)
  ... 12 more

The cause was that ServerCrashProcedure, written in some WAL file under MasterProcWALs from first run, was absent in 1.1 release.

After a brief discussion with Stephen, I am logging this JIRA to solicit discussion on how customer experience can be improved if downgrade of hbase is performed.

Issue Links

• is related to: HBASE-15100 Master WALProcs still never clean up