[HBASE-21150] Avoid delay in first flushes due to overheads in table metrics registration

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Ted Yu
• Created: 04/Sep/18 16:01
• Updated: 09/Sep/18 19:29
• Resolved: 05/Sep/18 16:22

Description

After HBASE-15728 is integrated, the lazy table metrics registration results in penalty for the first flushes.
Excerpt from log shows delay (note the same timestamp 08:18:23,234) :

2018-09-02 08:18:23,232 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableSourceImpl(124): Creating new                      MetricsTableSourceImpl for table 'testtb-1535901500805'
2018-09-02 08:18:23,233 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableSourceImpl(137): registering metrics for testtb-   1535901500805
2018-09-02 08:18:23,234 INFO  [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1] regionserver.HRegion(2822): Finished flush of dataSize ~2.29 KB/2343,       heapSize ~5.16 KB/5280, currentSize=0 B/0 for fa403f6a4fb8dbc1a1c389744fce2d58 in 280ms, sequenceid=5, compaction requested=false
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-1,5,FailOnTimeoutGroup]
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 0 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1,5,FailOnTimeoutGroup]
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register   testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-1,5,FailOnTimeoutGroup]
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register   testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-2,5,FailOnTimeoutGroup]
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 5 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-2,5,FailOnTimeoutGroup]
2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2,5,FailOnTimeoutGroup]

This is a regression in that there were multiple (6 ms) delays before the flush can finish, waiting for the metrics table to be registered.

When first region of the table is opened on region server, we can proactively register table metrics.
This would avoid the penalty on first flushes for the table.

Issue Links

• is broken by: HBASE-15728 Add remaining per-table region / store / flush / compaction related metrics