Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20610 Procedure V2 - Distributed Log Splitting
  3. HBASE-21588

Procedure v2 wal splitting implementation

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha-1, 2.2.0
    • 3.0.0-alpha-1, 2.2.0
    • None
    • None
    • Hide
      After HBASE-21588, we introduce a new way to do WAL splitting coordination by procedure framework. This can simplify the process of WAL splitting and no need to connect zookeeper any more.
      During ServerCrashProcedure, it will create a SplitWALProcedure for each WAL that needs to split. Then each SplitWALProcedure will spawn a SplitWALRemoteProcedure to send the request to regionserver.
      At the RegionServer side, whole split process is handled by a SplitWALCallable. It splits the WAL and returns the result to the master.
      According to my test, this patch has a better performance as the number of WALs that need to split increase. And it can relieve the pressure on zookeeper.

      Set 'hbase.split.wal.zk.coordinated' to false to 'enable'. 'hbase.regionserver.wal.max.splitters' is reused to set how many 'splitters' to run on the RegionServer side.
      Show
      After HBASE-21588 , we introduce a new way to do WAL splitting coordination by procedure framework. This can simplify the process of WAL splitting and no need to connect zookeeper any more. During ServerCrashProcedure, it will create a SplitWALProcedure for each WAL that needs to split. Then each SplitWALProcedure will spawn a SplitWALRemoteProcedure to send the request to regionserver. At the RegionServer side, whole split process is handled by a SplitWALCallable. It splits the WAL and returns the result to the master. According to my test, this patch has a better performance as the number of WALs that need to split increase. And it can relieve the pressure on zookeeper. Set 'hbase.split.wal.zk.coordinated' to false to 'enable'. 'hbase.regionserver.wal.max.splitters' is reused to set how many 'splitters' to run on the RegionServer side.

    Description

      create a sub task to submit the implementation of procedure v2 wal splitting

      Attachments

        1. HBASE-21588.branch-2.001.patch
          93 kB
          Jingyun Tian
        2. HBASE-21588.branch-2.002.patch
          93 kB
          Jingyun Tian
        3. HBASE-21588.master.003.patch
          93 kB
          Jingyun Tian
        4. HBASE-21588.master.004.patch
          93 kB
          Jingyun Tian
        5. HBASE-21588.master.005.patch
          93 kB
          Jingyun Tian
        6. HBASE-21588.master.006.patch
          84 kB
          Jingyun Tian
        7. HBASE-21588.master.007.patch
          84 kB
          Jingyun Tian
        8. HBASE-21588.master.008.patch
          82 kB
          Jingyun Tian
        9. HBASE-21588.master.009.patch
          95 kB
          Jingyun Tian
        10. HBASE-21588.master.010.patch
          87 kB
          Jingyun Tian
        11. HBASE-21588.master.011.patch
          87 kB
          Jingyun Tian
        12. HBASE-21588.master.012.patch
          86 kB
          Jingyun Tian
        13. HBASE-21588.master.013.patch
          87 kB
          Jingyun Tian
        14. HBASE-21588.master.014.patch
          85 kB
          Jingyun Tian
        15. HBASE-21588.master.015.patch
          87 kB
          Jingyun Tian
        16. HBASE-21588.master.016.patch
          87 kB
          Jingyun Tian
        17. HBASE-21588.master.017.patch
          87 kB
          Jingyun Tian
        18. HBASE-21588.master.018.patch
          88 kB
          Michael Stack
        19. HBASE-21588.master.018.patch
          88 kB
          Jingyun Tian
        20. HBASE-21588.master.019.patch
          89 kB
          Michael Stack
        21. HBASE-21588.master.019.patch
          89 kB
          Jingyun Tian
        22. HBASE-21588.master.020.patch
          89 kB
          Jingyun Tian

        Issue Links

        causes

        Bug - A problem which impairs or prevents the functions of the product. HBASE-25445 Old WALs archive fails in procedure based WAL split

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        is related to

        Sub-task - The sub-task of the issue HBASE-22153 Fix the flaky TestRestartCluster

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link Review Board (master)

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tianjingyun Jingyun Tian
            tianjingyun Jingyun Tian
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment