[HBASE-7235] TestMasterObserver is flaky

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.94.4, 0.95.2
• Fix Version/s: 0.94.4, 0.95.0
• Component/s: Coprocessors, master, test
• Hadoop Flags: Reviewed
• Assignee: Enis Soztutar
• Reporter: Enis Soztutar
• Created: 28/Nov/12 21:58
• Updated: 26/Feb/13 08:22
• Resolved: 29/Nov/12 00:16

Description

TestMasterObserver failed on windows builds at least a couple of times, although this is not windows specific.

Upon further inspection, it seems that we are affected by the race condition for the table operations.
The handling of modify table occurs in its dedicated executor, MASTER_TABLE_OPERATIONS, and happens in async, although the handling of add/delete/modify columns are handled in the IPC handler thread. When we do not wait for the first modify table to finish, subsequent changes from the add/modify column operations are overriden by the earlier modify table executor.

at org.apache.hadoop.hbase.master.handler.TableEventHandler.hasColumnFamily(TableEventHandler.java:182)
	at org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.<init>(TableDeleteFamilyHandler.java:42)
	at org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1255)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
	at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
	at org.apache.hadoop.hbase.client.HBaseAdmin.deleteColumn(HBaseAdmin.java:1024)
	at org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations(TestMasterObserver.java:616)