Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.20.0
    • Component/s: master, regionserver
    • Labels:
      None

      Description

      Currently, if a region server crashes and then restarts, it cannot be given work until its lease times out. This is because a lease is only identified by ipaddress:portnumber. If leases were also identified with the start code, the server could be given work immediately, because its log file includes the start code and will not interfere with the recovery of the log from its previous incarnation.

      Additionally, we wait in a master server thread for the server to leave the dead servers list because dead servers are not identified by their start code either. Waiting in a master server thread ties up that thread (possibly for quite some time), and rather than waiting, we should throw an exception as the region server already knows how to deal with an exception thrown from a regionServerStartup call.

      Finally, there is a bit of code cleanup that needs to be done in the region server when it receives a MSG_CALL_SERVER_STARTUP response from the master. It should not set up the HLog until reportForDuty completes
      successfully (which is what it does on the initial reportForDuty call.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          44d 1h 11m 1 Jim Kellerman 11/Mar/09 22:18
          In Progress In Progress Resolved Resolved
          2d 3h 20m 1 Jim Kellerman 14/Mar/09 01:39
          Resolved Resolved Closed Closed
          183d 20h 44m 1 stack 13/Sep/09 22:24
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jim Kellerman made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Jim Kellerman added a comment -

          Fixed as a part of HBASE-1157

          Show
          Jim Kellerman added a comment - Fixed as a part of HBASE-1157
          Jim Kellerman made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Jim Kellerman made changes -
          Parent HBASE-1157 [ 12413344 ]
          Issue Type Improvement [ 4 ] Sub-task [ 7 ]
          Jim Kellerman made changes -
          Link This issue blocks HBASE-1144 [ HBASE-1144 ]
          Jim Kellerman made changes -
          Link This issue is related to HBASE-1157 [ HBASE-1157 ]
          Jim Kellerman made changes -
          Field Original Value New Value
          Link This issue relates to HBASE-1158 [ HBASE-1158 ]
          Hide
          Jim Kellerman added a comment -

          If we included the start code in the check, then the restarted server could start serving regions immediately without its regions being detected as having been hosted by the dead server.

          Show
          Jim Kellerman added a comment - If we included the start code in the check, then the restarted server could start serving regions immediately without its regions being detected as having been hosted by the dead server.
          Hide
          Jim Kellerman added a comment -

          It turns out that currently there is a good reason for not allowing a restarted server to rejoin the cluster until ProcessServerShutdown is complete: we don't check the start code, so any server instance from the same ip:port pair will match. Ugh! Because we don't check the start-code, we cannot allow a server to start serving regions until ProcessServerShutdown is complete. If we checked the start code, we would know if a region was on the dead server and we could reassign it. Otherwise, we might end up reassigning regions being served by the new instance, resulting in multiple server serving the same region and possibly before the log had been recovered.

          Show
          Jim Kellerman added a comment - It turns out that currently there is a good reason for not allowing a restarted server to rejoin the cluster until ProcessServerShutdown is complete: we don't check the start code, so any server instance from the same ip:port pair will match. Ugh! Because we don't check the start-code, we cannot allow a server to start serving regions until ProcessServerShutdown is complete. If we checked the start code, we would know if a region was on the dead server and we could reassign it. Otherwise, we might end up reassigning regions being served by the new instance, resulting in multiple server serving the same region and possibly before the log had been recovered.
          Jim Kellerman created issue -

            People

            • Assignee:
              Jim Kellerman
              Reporter:
              Jim Kellerman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development