Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.20.0
    • Component/s: master, regionserver
    • Labels:
      None

      Description

      Currently, if a region server crashes and then restarts, it cannot be given work until its lease times out. This is because a lease is only identified by ipaddress:portnumber. If leases were also identified with the start code, the server could be given work immediately, because its log file includes the start code and will not interfere with the recovery of the log from its previous incarnation.

      Additionally, we wait in a master server thread for the server to leave the dead servers list because dead servers are not identified by their start code either. Waiting in a master server thread ties up that thread (possibly for quite some time), and rather than waiting, we should throw an exception as the region server already knows how to deal with an exception thrown from a regionServerStartup call.

      Finally, there is a bit of code cleanup that needs to be done in the region server when it receives a MSG_CALL_SERVER_STARTUP response from the master. It should not set up the HLog until reportForDuty completes
      successfully (which is what it does on the initial reportForDuty call.

        Issue Links

          Activity

          Hide
          Jim Kellerman added a comment -

          Fixed as a part of HBASE-1157

          Show
          Jim Kellerman added a comment - Fixed as a part of HBASE-1157
          Hide
          Jim Kellerman added a comment -

          If we included the start code in the check, then the restarted server could start serving regions immediately without its regions being detected as having been hosted by the dead server.

          Show
          Jim Kellerman added a comment - If we included the start code in the check, then the restarted server could start serving regions immediately without its regions being detected as having been hosted by the dead server.
          Hide
          Jim Kellerman added a comment -

          It turns out that currently there is a good reason for not allowing a restarted server to rejoin the cluster until ProcessServerShutdown is complete: we don't check the start code, so any server instance from the same ip:port pair will match. Ugh! Because we don't check the start-code, we cannot allow a server to start serving regions until ProcessServerShutdown is complete. If we checked the start code, we would know if a region was on the dead server and we could reassign it. Otherwise, we might end up reassigning regions being served by the new instance, resulting in multiple server serving the same region and possibly before the log had been recovered.

          Show
          Jim Kellerman added a comment - It turns out that currently there is a good reason for not allowing a restarted server to rejoin the cluster until ProcessServerShutdown is complete: we don't check the start code, so any server instance from the same ip:port pair will match. Ugh! Because we don't check the start-code, we cannot allow a server to start serving regions until ProcessServerShutdown is complete. If we checked the start code, we would know if a region was on the dead server and we could reassign it. Otherwise, we might end up reassigning regions being served by the new instance, resulting in multiple server serving the same region and possibly before the log had been recovered.

            People

            • Assignee:
              Jim Kellerman
              Reporter:
              Jim Kellerman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development