[PHOENIX-3698] Missing no-args constructor on IndexedWALEditCodec

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.9.1, 4.10.0
• Assignee: Josh Elser
• Reporter: Romil Choksi
• Created: 27/Feb/17 21:54
• Updated: 01/Mar/17 04:41
• Resolved: 01/Mar/17 00:56

Description

From a deploy at $dayjob, I'm seeing what I'm think is a pretty glaring omission in the WAL splitting path with the IndexedWALEditCodec. "Pretty glaring" to the degree that I think I might just be missing something...

2017-02-23 09:05:05,652 ERROR [RS_LOG_REPLAY_OPS-10.0.0.13:16020-0-Writer-1] wal.WALSplitter: Exiting thread
java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
	at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:430)
	at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:423)
	at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:301)
	at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:116)
	at org.apache.hadoop.hbase.wal.WALSplitter$LogReplayOutputSink.getConnectionByTableName(WALSplitter.java:2145)
	at org.apache.hadoop.hbase.wal.WALSplitter$LogReplayOutputSink.groupEditsByServer(WALSplitter.java:1750)
	at org.apache.hadoop.hbase.wal.WALSplitter$LogReplayOutputSink.append(WALSplitter.java:1690)
	at org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1104)
	at org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1096)
	at org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1066)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
	... 10 more
Caused by: java.lang.UnsupportedOperationException: Constructor threw an exception for org.apache.hadoop.hbase.ipc.RpcClientImpl
	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:54)
	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:34)
	at org.apache.hadoop.hbase.ipc.RpcClientFactory.createClient(RpcClientFactory.java:64)
	at org.apache.hadoop.hbase.ipc.RpcClientFactory.createClient(RpcClientFactory.java:48)
	at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:647)
	... 15 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:46)
	... 19 more
Caused by: java.lang.RuntimeException: Failed getting codec org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getCodec(AbstractRpcClient.java:138)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.<init>(AbstractRpcClient.java:95)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.<init>(RpcClientImpl.java:1082)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.<init>(RpcClientImpl.java:1108)
	... 24 more
Caused by: java.lang.InstantiationException: org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec
	at java.lang.Class.newInstance(Class.java:427)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.getCodec(AbstractRpcClient.java:136)
	... 27 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.<init>()
	at java.lang.Class.getConstructor0(Class.java:3082)
	at java.lang.Class.newInstance(Class.java:412)
	... 28 more

Essentially, our IndexedWALEditCodec has the potential to be constructed in two different code paths: As a Codec and as a WALCellCodec.

The latter is what we have accounted for, the Configuration and CompressionContext constructor. The former is the one we don't account for, missing a no-args constructor.

WALSplitter will set the value for a custom WAL codec as the client RPC codec in the configuration when it's creating a Connection.

https://github.com/apache/hbase/blob/rel/1.2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java#L184-L186

This caused a cascading failure across all of the RegionServers.