[HBASE-21349] Cluster is going down but CatalogJanitor and Normalizer try to run and fail noisely

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.3
• Hadoop Flags: Reviewed
• Assignee: Xu Cang
• Reporter: Michael Stack
• Created: 20/Oct/18 04:37
• Updated: 24/Oct/18 13:39
• Resolved: 23/Oct/18 21:45

Description

Shutting down can take a while. Meantime catalog janitor and or normalizer (etc?) try to run and when they can't, they fail noisely. Looks bad:

2018-10-19 21:23:24,962 INFO org.apache.hadoop.hbase.master.ServerManager: Cluster shutdown set; vc1205.halxg.cloudera.com,22101,1539991730711 expired; onlineServers=51
2018-10-19 21:25:54,502 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog table
java.io.IOException: connection is closed
        at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:684)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMetaForTableRegions(MetaTableAccessor.java:679)
        at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:185)
        at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:137)
        at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:243)
        at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:116)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-19 21:25:54,507 ERROR org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to normalize regions.
java.io.IOException: connection is closed
        at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
        at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690)
        at org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:240)
        at org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
        at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1718)
        at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
        at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)