Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-6134

Improvement for split-worker to speed up distributed log splitting

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.95.0
    • Component/s: wal
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      First,we do the test between local-master-splitting and distributed-log-splitting

      Environment:34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths splitting work), 400 regions in one hlog file

      local-master-split:60s+
      distributed-log-splitting:165s+

      In fact, in our production environment, distributed-log-splitting also took 60s with 30 regionservers for 34 hlog files (regionserver may be in high load)

      We found split-worker split one log file took about 20s
      (30ms~50ms per writer.close(); 10ms per create writers )

      I think we could do the improvement for this:
      Parallelizing the create and close writers in threads

      In the patch, change the logic for distributed-log-splitting same as the local-master-splitting and parallelizing the close in threads.

        Attachments

        1. 6134v4.patch
          21 kB
          Ted Yu
        2. HBASE-6134.patch
          18 kB
          Chunhui Shen
        3. HBASE-6134v2.patch
          19 kB
          Chunhui Shen
        4. HBASE-6134v3.patch
          21 kB
          Chunhui Shen
        5. HBASE-6134v3-92.patch
          22 kB
          Chunhui Shen
        6. HBASE-6134v4.patch
          21 kB
          Chunhui Shen
        7. HBASE-6134v4-94.patch
          21 kB
          Chunhui Shen
        8. HBASE-6134v4-94v2.patch
          22 kB
          Sergey Shelukhin

          Issue Links

            Activity

              People

              • Assignee:
                zjushch Chunhui Shen
                Reporter:
                zjushch Chunhui Shen
              • Votes:
                0 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: