[PHOENIX-7076] MetaDataRegionObserver#postOpen hook improvements

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.1.3
• Fix Version/s: 5.2.0, 5.1.4
• Assignee: Divneet Kaur
• Reporter: Viraj Jasani
• Created: 18/Oct/23 23:13
• Updated: 13/Dec/23 04:20
• Resolved: 13/Dec/23 04:20

Description

For older versions upgrading to 4.7+, we perform deletion of all rows on SYSTEM.STATS table one time. This is done only if column "UPGRADE_TO_4_7" does not exist on SYSTEM.CATALOG.

Since this task is performed in a daemon thread as part of region#postOpen coproc hook, it might get stuck accessing SYSTEM.CATALOG table if the region open gets delayed for some reason.

Similar to BuildIndexScheduleTask that gets scheduled with 10s of initial delay (by default), we should perform truncate stats as a delayed task, even though it does not require any repetition.

 

2023-10-11 22:51:09,728 DEBUG [Thread-60] client.RpcRetryingCallerImpl - n$ClientServiceBlockingInterfaceWrapper.doCall(ServerConnectionUtils.java:163) 
    at org.apache.hadoop.hbase.client.ServerConnectionUtils$ShortCircuitingClusterConnection$ClientServiceBlockingInterfaceWrapper.get(ServerConnectionUtils.java:131) 
    at org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:51) 
    at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:364) 
    at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:359) 
    at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) 
    at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) 
    at org.apache.hadoop.hbase.client.HTable.get(HTable.java:370) 
    at org.apache.hadoop.hbase.client.HTable.get(HTable.java:343) 
    at org.apache.phoenix.util.UpgradeUtil.truncateStats(UpgradeUtil.java:2057) 
    at org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:182) 
    at org.apache.phoenix.coprocessor.MetaDataRegionObserver$1$1.run(MetaDataRegionObserver.java:179) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:422) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962) 
    at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:514) 
    at org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:495) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39) 
    at org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:178) 
    at org.apache.phoenix.coprocessor.MetaDataRegionObserver$1.run(MetaDataRegionObserver.java:179) 
    at java.lang.Thread.run(Thread.java:750) 

 

Improvements:

1. Schedule truncate stats task with 10s delay (configurable) rather than starting the thread immediately.
2. Only if phoenix.stats.collection.enabled is enabled, create daemon thread for truncate stats task.

Issue Links

• links to: GitHub Pull Request #1735
• links to: GitHub Pull Request #1757