[HBASE-15752] ClassNotFoundException is encountered when custom WAL codec is not found in WALPlayer job

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.0, 0.98.20, 2.0.0, 1.2.10
• Component/s: tooling, wal
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 02/May/16 22:05
• Updated: 07/Jan/19 21:30
• Resolved: 03/May/16 14:02

Description

cartershanklin reported the following when he tried out back / restore feature in a Phoenix enabled deployment:

2016-05-02 18:57:58,578 FATAL [IPC Server handler 2 on 38194] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1462215011294_0001_m_000000_0 - exited : java.io. IOException: Cannot get log reader
  at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:344)
  at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:266)
  at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:254)
  at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:403)
  at org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.initialize(WALInputFormat.java:152)
  at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1708)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
  at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36)
  at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.create(WALCellCodec.java:103)
  at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.getCodec(ProtobufLogReader.java:282)
  at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
  at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
  at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:149)
  at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:301)
  ... 12 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:264)

This was due to the IndexedWALEditCodec (specified thru hbase.regionserver.wal.codec) used by Phoenix being absent in hadoop classpath.

WALPlayer should handle this situation better by adding the jar for IndexedWALEditCodec class to mapreduce job dependency.

Although this was found during testing of backup / restore, the error may occur in other places where WALPlayer needs custom WAL codec for the replay.