Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21150

Avoid delay in first flushes due to overheads in table metrics registration

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      After HBASE-15728 is integrated, the lazy table metrics registration results in penalty for the first flushes.
      Excerpt from log shows delay (note the same timestamp 08:18:23,234) :

      2018-09-02 08:18:23,232 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableSourceImpl(124): Creating new                      MetricsTableSourceImpl for table 'testtb-1535901500805'
      2018-09-02 08:18:23,233 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableSourceImpl(137): registering metrics for testtb-   1535901500805
      2018-09-02 08:18:23,234 INFO  [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1] regionserver.HRegion(2822): Finished flush of dataSize ~2.29 KB/2343,       heapSize ~5.16 KB/5280, currentSize=0 B/0 for fa403f6a4fb8dbc1a1c389744fce2d58 in 280ms, sequenceid=5, compaction requested=false
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-1,5,FailOnTimeoutGroup]
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 0 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-1,5,FailOnTimeoutGroup]
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-1] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register   testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-1,5,FailOnTimeoutGroup]
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register   testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52762,1535901497314)-snapshot-pool9-thread-2,5,FailOnTimeoutGroup]
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 5 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52758,1535901497238)-snapshot-pool11-thread-2,5,FailOnTimeoutGroup]
      2018-09-02 08:18:23,234 DEBUG [rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2] regionserver.MetricsTableAggregateSourceImpl(84): it took 6 ms to register  testtb-1535901500805 Thread[rs(hw13463.attlocal.net,52760,1535901497280)-snapshot-pool10-thread-2,5,FailOnTimeoutGroup]
      

      This is a regression in that there were multiple (6 ms) delays before the flush can finish, waiting for the metrics table to be registered.

      When first region of the table is opened on region server, we can proactively register table metrics.
      This would avoid the penalty on first flushes for the table.

      Attachments

        1. 21150.v1.txt
          8 kB
          Ted Yu
        2. 21150.v2.txt
          8 kB
          Ted Yu
        3. 21150.v3.txt
          8 kB
          Ted Yu
        4. 21150.v4.txt
          7 kB
          Ted Yu
        5. 21150.v4.txt
          7 kB
          Ted Yu
        6. 21150.v5.txt
          6 kB
          Ted Yu

        Issue Links

          Activity

            People

              Unassigned Unassigned
              yuzhihong@gmail.com Ted Yu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: