[HBASE-25166] MobFileCompactionChore is closing the master's shared cluster connection

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 2.6.0, 2.5.1
• Component/s: master
• Assignee: Ankit Singhal
• Reporter: Ankit Singhal
• Created: 08/Oct/20 16:48
• Updated: 14/Oct/22 18:59
• Resolved: 20/Oct/20 05:24

Description

Code which is doing so in MobFileCompactionChore

try (Connection conn = master.getConnection();
    Admin admin = conn.getAdmin();) { 

As master uses this connection to read the meta or other system tables, so none of the meta operations through master will work.

Symptoms in master logs:-

s, events=841, succcessCount=123, totalEvents=12824192, totalSuccessCount=1891300
2020-10-05 16:34:25,062 INFO org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: Unable to get remote Address
2020-10-05 16:34:25,062 ERROR org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to normalize regions.
java.io.IOException: connection is closed
        at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:797)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:768)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:727)
        at org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:215)
        at org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
        at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1821)
        at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:188)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748) 

Symptoms at the client:-

 RpcRetryingCaller{globalStartTime=1602099132430, pause=100, maxAttempts=11}, java.io.IOException: java.io.IOException: connection is closed
       at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:241)
       at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1116)
       at org.apache.hadoop.hbase.master.TableStateManager.readMetaState(TableStateManager.java:258)
       at org.apache.hadoop.hbase.master.TableStateManager.isTablePresent(TableStateManager.java:175)
       at org.apache.hadoop.hbase.master.HMaster.getTableDescriptors(HMaster.java:3277)
       at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:3221)
       at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1064)
       at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:418)
       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
       at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318

Issue Links

• is caused by: HBASE-22749 Distributed MOB compactions
• is related to: HBASE-24686 [LOG] Log improvement in Connection#close
• links to: GitHub Pull Request #2514
• links to: GitHub Pull Request #4825
• links to: GitHub Pull Request #4826