XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: proc-v2
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Replaces the AssignmentManager with a new procedurev2-based AssignmentManager

      h1. AMv2
      Puts AssignmentManager up on top of the ProcedureV2 state machine with persistence engine. Each assignment atom is now a Procedure implementation; e.g. an AssignProcedure and an UnassignProcedure. Molecules of aggregated Procedures are used to do more involved assignment steps: e.g. the move region procedure is made of an Unassign followed by an Assign subprocedure.

      AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to Procedures. In-memory states of regions and servers has been cleaned up stored in new RegionStates implementation. RegionStateStore takes care of publishing final region state out to the hbase:meta table.

      New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based assignments ‘remotely’. Knows about ‘servers’. Does aggregation of assignments by time on a time/count basis so can send procedures in batches rather than one per RPC. Procedure status comes back on the back of the RegionServer heartbeat reporting online regions. The response is passed to the AMv2 to ‘process’. It will check against the in-memory state. If there is a mismatch, it fences out the RegionServer on the assumption that something went wrong on the RS side.Timeouts trigger retries. The Procedure machine ensures only one operation at a time on any one region/table using locking and smarts about what is serial and what can be run concurrently.

      New accounting of RegionServer version will be used running rolling restarts.

      ‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only serialized out to the ProcedureV2 WAL. They are no longer persisted to ZooKeeper.

      h2. Assign Detail
      The Assign starts by pushing the "assign" operation to the AssignmentManager and then will go into a “waiting" state. The AM will batch the "assign" requests and ask the Balancer where to put the region (the various policies will be respected: retain, round-robin, random). Once the AM and the balancer have found a place for the region, the procedure will be resumed and an "open region" request will be placed in the Remote Dispatcher queue, and the procedure once again will go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the assignment by publishing to new state on hbase:meta or it will retry the assignment.

      h3. Unassign Detail
       The Unassign starts by placing a "close region" request in the Remote Dispatcher queue, and the procedure will then go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the unassign by publishing its new state on meta or it will retry the unassign.

      h1. New Configs
       * "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
       * "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
       * "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
       * "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
      h1. TODO
      As of this writing.

      Put up a model diagram.

       * Handle region migration
       * Handle meta assignment first
       * Handle sys table assignment first (e.g. acl, namespace)
       * Handle table priorities
       * Do we report same AM metrics as we used too? We do it all in here now.

      INCOMPATIBLE
      A known incompatible is that because splits and merges are now run from the master, Coprocessors that used to watch for merge/split from a RegionObserver now no longer work; to watch split/merges, you need to have an observer on the Master instead.
      Show
      Replaces the AssignmentManager with a new procedurev2-based AssignmentManager h1. AMv2 Puts AssignmentManager up on top of the ProcedureV2 state machine with persistence engine. Each assignment atom is now a Procedure implementation; e.g. an AssignProcedure and an UnassignProcedure. Molecules of aggregated Procedures are used to do more involved assignment steps: e.g. the move region procedure is made of an Unassign followed by an Assign subprocedure. AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to Procedures. In-memory states of regions and servers has been cleaned up stored in new RegionStates implementation. RegionStateStore takes care of publishing final region state out to the hbase:meta table. New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based assignments ‘remotely’. Knows about ‘servers’. Does aggregation of assignments by time on a time/count basis so can send procedures in batches rather than one per RPC. Procedure status comes back on the back of the RegionServer heartbeat reporting online regions. The response is passed to the AMv2 to ‘process’. It will check against the in-memory state. If there is a mismatch, it fences out the RegionServer on the assumption that something went wrong on the RS side.Timeouts trigger retries. The Procedure machine ensures only one operation at a time on any one region/table using locking and smarts about what is serial and what can be run concurrently. New accounting of RegionServer version will be used running rolling restarts. ‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only serialized out to the ProcedureV2 WAL. They are no longer persisted to ZooKeeper. h2. Assign Detail The Assign starts by pushing the "assign" operation to the AssignmentManager and then will go into a “waiting" state. The AM will batch the "assign" requests and ask the Balancer where to put the region (the various policies will be respected: retain, round-robin, random). Once the AM and the balancer have found a place for the region, the procedure will be resumed and an "open region" request will be placed in the Remote Dispatcher queue, and the procedure once again will go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the assignment by publishing to new state on hbase:meta or it will retry the assignment. h3. Unassign Detail  The Unassign starts by placing a "close region" request in the Remote Dispatcher queue, and the procedure will then go into a "waiting state". The Remote Dispatcher will batch the various requests for that server and they will be sent to the RS for execution. The RS will complete the open operation by calling master.reportRegionStateTransition(). The AM will intercept the transition report, and notify the procedure. The procedure will finish the unassign by publishing its new state on meta or it will retry the unassign. h1. New Configs  * "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128  * "hbase.procedure.remote.dispatcher.delay.msec" default 150ms  * "hbase.procedure.remote.dispatcher.max.queue.size" with default 32  * "hbase.regionserver.rpc.startup.waittime" with default 60 seconds. h1. TODO As of this writing. Put up a model diagram.  * Handle region migration  * Handle meta assignment first  * Handle sys table assignment first (e.g. acl, namespace)  * Handle table priorities  * Do we report same AM metrics as we used too? We do it all in here now. INCOMPATIBLE A known incompatible is that because splits and merges are now run from the master, Coprocessors that used to watch for merge/split from a RegionObserver now no longer work; to watch split/merges, you need to have an observer on the Master instead.

      Description

      New AssignmentManager implemented using proc-v2.

      • AssignProcedure handle assignment operation
      • UnassignProcedure handle unassign operation
      • MoveRegionProcedure handle move/balance operation

      Concurrent Assign operations are batched together and sent to the balancer
      Concurrent Assign and Unassign operation ready to be sent to the RS are batched together

      This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of HBASE-14616

        Attachments

        1. HBASE-14614.master.003.patch
          1.04 MB
          Michael Stack
        2. HBASE-14614.master.004.patch
          1.05 MB
          Michael Stack
        3. HBASE-14614.master.005.patch
          1.06 MB
          Michael Stack
        4. HBASE-14614.master.006.patch
          1.10 MB
          Michael Stack
        5. HBASE-14614.master.007.patch
          1.08 MB
          Michael Stack
        6. HBASE-14614.master.008.patch
          1.08 MB
          Michael Stack
        7. HBASE-14614.master.009.patch
          1.16 MB
          Michael Stack
        8. HBASE-14614.master.010.patch
          1.17 MB
          Michael Stack
        9. HBASE-14614.master.014.patch
          1.16 MB
          Michael Stack
        10. HBASE-14614.master.015.patch
          1.16 MB
          Michael Stack
        11. HBASE-14614.master.017.patch
          3.52 MB
          Michael Stack
        12. HBASE-14614.master.018.patch
          3.53 MB
          Michael Stack
        13. HBASE-14614.master.019.patch
          3.55 MB
          Michael Stack
        14. HBASE-14614.master.020.patch
          3.55 MB
          Michael Stack
        15. HBASE-14614.master.022.patch
          3.55 MB
          Michael Stack
        16. HBASE-14614.master.023.patch
          3.57 MB
          Michael Stack
        17. HBASE-14614.master.024.patch
          3.57 MB
          Michael Stack
        18. HBASE-14614.master.025.patch
          3.38 MB
          Michael Stack
        19. HBASE-14614.master.026.patch
          3.39 MB
          Michael Stack
        20. HBASE-14614.master.027.patch
          3.39 MB
          Michael Stack
        21. HBASE-14614.master.028.patch
          3.41 MB
          Michael Stack
        22. HBASE-14614.master.029.patch
          3.42 MB
          Michael Stack
        23. HBASE-14614.master.030.patch
          3.42 MB
          Michael Stack
        24. HBASE-14614.master.033.patch
          4.08 MB
          Michael Stack
        25. HBASE-14614.master.038.patch
          3.47 MB
          Michael Stack
        26. HBASE-14614.master.039.patch
          3.51 MB
          Michael Stack
        27. HBASE-14614.master.040.patch
          4.08 MB
          Michael Stack
        28. HBASE-14614.master.041.patch
          4.09 MB
          Michael Stack
        29. HBASE-14614.master.042.patch
          4.08 MB
          Michael Stack
        30. HBASE-14614.master.043.patch
          4.08 MB
          Michael Stack
        31. HBASE-14614.master.044.patch
          4.08 MB
          Michael Stack
        32. HBASE-14614.master.045.patch
          4.08 MB
          Michael Stack
        33. HBASE-14614.master.045.patch
          4.08 MB
          Michael Stack
        34. HBASE-14614.master.046.patch
          4.10 MB
          Michael Stack
        35. HBASE-14614.master.047.patch
          4.10 MB
          Michael Stack
        36. HBASE-14614.master.048.patch
          4.10 MB
          Michael Stack
        37. HBASE-14614.master.049.patch
          4.11 MB
          Michael Stack
        38. HBASE-14614.master.050.patch
          4.11 MB
          Michael Stack
        39. HBASE-14614.master.051.patch
          4.11 MB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                mbertozzi Matteo Bertozzi
                Reporter:
                syuanjiang Stephen Yuan Jiang
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: