[HBASE-9822] IntegrationTestLazyCfLoading failed occasionally in a secure enviroment

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 0.96.0
• Fix Version/s: 0.98.0, 0.96.1
• Component/s: test
• Hadoop Flags: Reviewed
• Assignee: Jeffrey Zhong
• Reporter: Jeffrey Zhong
• Created: 22/Oct/13 22:54
• Updated: 16/Dec/13 18:46
• Resolved: 30/Oct/13 18:50

Description

This test case failed in a secure deployment once with the following error. It's due to a race condition between writers starts writes and table ACLs propagation to region servers.

2013-10-14 13:03:32,185 ERROR [HBaseWriterThread_8] util.MultiThreadedWriterBase: Failed to insert: 10 after 167ms; region information: cached: region=IntegrationTestLazyCfLoading,bffffffd,1381755808862.456a11d22693f7dc27763c32e55521a8., hostname=gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694, seqNum=1; cache is up to date; errors: exception from gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal:60020 for d3d9446802a44259755d38e6d163e820-10
E       org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (table=IntegrationTestLazyCfLoading, family: essential:filter, action=WRITE)
....

Writes were sent at 13:03:32,032

2013-10-14 13:03:32,032 WARN  [htable-pool11-t1] client.AsyncProcess: Attempt #1/35 failed for 1 ops on gs-hdp2-secure-1381732260-hbase-8.cs1cloud.internal,60020,1381755752694 NOT resubmitting.

While the permission propagation happened at 13:03:32,109 on region server

2013-10-14 13:03:32,109 DEBUG [regionserver60020-EventThread] access.ZKPermissionWatcher: Updating permissions cache from node IntegrationTestLazyCfLoading with data: PBUF\x0AA\x0A\x06hrt_qa\x127\x08\x03"3\x0A'\x0A\x07default\x12\x1CIntegrationTestLazyCfLoading \x00 \x01 \x02 \x03 \x04