Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20610 Procedure V2 - Distributed Log Splitting
  3. HBASE-21588

Procedure v2 wal splitting implementation

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha-1, 2.2.0
    • 3.0.0-alpha-1, 2.2.0
    • None
    • None
    • Hide
      After HBASE-21588, we introduce a new way to do WAL splitting coordination by procedure framework. This can simplify the process of WAL splitting and no need to connect zookeeper any more.
      During ServerCrashProcedure, it will create a SplitWALProcedure for each WAL that needs to split. Then each SplitWALProcedure will spawn a SplitWALRemoteProcedure to send the request to regionserver.
      At the RegionServer side, whole split process is handled by a SplitWALCallable. It splits the WAL and returns the result to the master.
      According to my test, this patch has a better performance as the number of WALs that need to split increase. And it can relieve the pressure on zookeeper.

      Set 'hbase.split.wal.zk.coordinated' to false to 'enable'. 'hbase.regionserver.wal.max.splitters' is reused to set how many 'splitters' to run on the RegionServer side.
      Show
      After HBASE-21588 , we introduce a new way to do WAL splitting coordination by procedure framework. This can simplify the process of WAL splitting and no need to connect zookeeper any more. During ServerCrashProcedure, it will create a SplitWALProcedure for each WAL that needs to split. Then each SplitWALProcedure will spawn a SplitWALRemoteProcedure to send the request to regionserver. At the RegionServer side, whole split process is handled by a SplitWALCallable. It splits the WAL and returns the result to the master. According to my test, this patch has a better performance as the number of WALs that need to split increase. And it can relieve the pressure on zookeeper. Set 'hbase.split.wal.zk.coordinated' to false to 'enable'. 'hbase.regionserver.wal.max.splitters' is reused to set how many 'splitters' to run on the RegionServer side.

    Description

      create a sub task to submit the implementation of procedure v2 wal splitting

      Attachments

        1. HBASE-21588.master.003.patch
          93 kB
          Jingyun Tian
        2. HBASE-21588.master.004.patch
          93 kB
          Jingyun Tian
        3. HBASE-21588.master.005.patch
          93 kB
          Jingyun Tian
        4. HBASE-21588.master.006.patch
          84 kB
          Jingyun Tian
        5. HBASE-21588.master.007.patch
          84 kB
          Jingyun Tian
        6. HBASE-21588.master.008.patch
          82 kB
          Jingyun Tian
        7. HBASE-21588.master.009.patch
          95 kB
          Jingyun Tian
        8. HBASE-21588.master.010.patch
          87 kB
          Jingyun Tian
        9. HBASE-21588.master.011.patch
          87 kB
          Jingyun Tian
        10. HBASE-21588.master.012.patch
          86 kB
          Jingyun Tian
        11. HBASE-21588.master.013.patch
          87 kB
          Jingyun Tian
        12. HBASE-21588.master.014.patch
          85 kB
          Jingyun Tian
        13. HBASE-21588.master.015.patch
          87 kB
          Jingyun Tian
        14. HBASE-21588.master.016.patch
          87 kB
          Jingyun Tian
        15. HBASE-21588.master.017.patch
          87 kB
          Jingyun Tian
        16. HBASE-21588.master.018.patch
          88 kB
          Jingyun Tian
        17. HBASE-21588.master.018.patch
          88 kB
          Michael Stack
        18. HBASE-21588.master.019.patch
          89 kB
          Jingyun Tian
        19. HBASE-21588.master.019.patch
          89 kB
          Michael Stack
        20. HBASE-21588.master.020.patch
          89 kB
          Jingyun Tian
        21. HBASE-21588.branch-2.001.patch
          93 kB
          Jingyun Tian
        22. HBASE-21588.branch-2.002.patch
          93 kB
          Jingyun Tian

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. HBASE-25445 Old WALs archive fails in procedure based WAL split

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HBASE-22153 Fix the flaky TestRestartCluster

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link Review Board (master)

          Activity

            People

              tianjingyun Jingyun Tian
              tianjingyun Jingyun Tian
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: