Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: General
    • Labels:
      None

      Description

      HBase init scripts, like all Hadoop init scripts, are killed at the same time (K20-*) in /etc/rc0.d and /etc/rc6.d.

      If the HMaster and/or the datanode shuts down before the regionserver, this causes the regionserver to take a long time to actually give up and shut down, effectively blocking system shutdown. This is a big problem on my pseudo-distributed instance on my laptop.

      The init script default stops for HBase services should be set to something like:

      K17hadoop-hbase-regionserver
      K18hadoop-hbase-master
      K19hadoop-zookeeper-server
      K20hadoop-*

      1. BIGTOP-239.patch
        6 kB
        Plamen Jeliazkov
      2. BIGTOP-239.patch
        3 kB
        Plamen Jeliazkov
      3. BIGTOP-239.patch
        3 kB
        Plamen Jeliazkov
      4. BIGTOP-239.patch
        6 kB
        Plamen Jeliazkov
      5. BIGTOP-239.patch
        6 kB
        Plamen Jeliazkov

        Activity

        Hide
        Konstantin Boudnik added a comment -

        Committed to trunk. Thanks Plamen, congrats on your first contribution!

        Show
        Konstantin Boudnik added a comment - Committed to trunk. Thanks Plamen, congrats on your first contribution!
        Hide
        Bruno Mahé added a comment -

        Plamen> I also just added you to the contributors group so you can be marked as assignee.

        Show
        Bruno Mahé added a comment - Plamen> I also just added you to the contributors group so you can be marked as assignee.
        Hide
        Bruno Mahé added a comment -

        +1 from me as well.
        Thanks a lot!

        Show
        Bruno Mahé added a comment - +1 from me as well. Thanks a lot!
        Hide
        Konstantin Boudnik added a comment -

        Looks about right. +1 from my limited knowledge in the matter. Any other takers? Otherwise I will commit it tomorrow.

        Show
        Konstantin Boudnik added a comment - Looks about right. +1 from my limited knowledge in the matter. Any other takers? Otherwise I will commit it tomorrow.
        Hide
        Plamen Jeliazkov added a comment -

        There was actually a mistake in that last patch. I removed a segment of coding that chose between the SUSE .sh file and the other .sh file in the .spec. Uploading new patch – this should address all 3 points now. Review would be appreciated. I will also run another HBase RPM creation to verify.

        Show
        Plamen Jeliazkov added a comment - There was actually a mistake in that last patch. I removed a segment of coding that chose between the SUSE .sh file and the other .sh file in the .spec. Uploading new patch – this should address all 3 points now. Review would be appreciated. I will also run another HBase RPM creation to verify.
        Hide
        Plamen Jeliazkov added a comment -

        Latest patch addresses points 1 & 2.

        Show
        Plamen Jeliazkov added a comment - Latest patch addresses points 1 & 2.
        Hide
        Plamen Jeliazkov added a comment -

        Correction to my #2) I can see the start sequence in .spec as:
        %service_macro master
        %service_macro thrift
        %service_macro regionserver
        %service_macro rest

        I will order the chkconfig to start in that order and stop in the reverse order. I will also still add a *) case to set anything else start after those.

        Show
        Plamen Jeliazkov added a comment - Correction to my #2) I can see the start sequence in .spec as: %service_macro master %service_macro thrift %service_macro regionserver %service_macro rest I will order the chkconfig to start in that order and stop in the reverse order. I will also still add a *) case to set anything else start after those.
        Hide
        Plamen Jeliazkov added a comment -

        Thanks for the review Sean! So should I still along the lines of the most recent patch then?

        Responses below:
        1) Okay, that makes setting up chkconfig even easier then!
        2) I figure rest and thrift services can be added as a *) case and set to the same chkconfig – I am pretty confident they can just be brought down at the same time before RegionServer and Master shutdown. I can make a patch towards that. Thanks for the catch!
        3) I am unfamiliar with this template but I will take a look. Should that be a separate JIRA or is it part of this one?

        Show
        Plamen Jeliazkov added a comment - Thanks for the review Sean! So should I still along the lines of the most recent patch then? Responses below: 1) Okay, that makes setting up chkconfig even easier then! 2) I figure rest and thrift services can be added as a *) case and set to the same chkconfig – I am pretty confident they can just be brought down at the same time before RegionServer and Master shutdown. I can make a patch towards that. Thanks for the catch! 3) I am unfamiliar with this template but I will take a look. Should that be a separate JIRA or is it part of this one?
        Hide
        Sean Mackrory added a comment -

        Just saw the most recent patch. A few thoughts from me:

        • The runlevels should be 345 on Red Hat as well. I believe it's a mistake that they were 2345 previously. Runlevel 2 is networkless (except on Debian / Ubuntu).
        • hbase_services is more than just master and regionserver - we also need to set chkconfig for for the rest and thrift services.
        • We should also apply this fix for deb/service-init.d.tpl.
        Show
        Sean Mackrory added a comment - Just saw the most recent patch. A few thoughts from me: The runlevels should be 345 on Red Hat as well. I believe it's a mistake that they were 2345 previously. Runlevel 2 is networkless (except on Debian / Ubuntu). hbase_services is more than just master and regionserver - we also need to set chkconfig for for the rest and thrift services. We should also apply this fix for deb/service-init.d.tpl.
        Hide
        Sean Mackrory added a comment -

        I think using .svc files is a good idea - but as you pointed out, there are differences between hbase.sh and hbase.sh.suse. DEB packages are using an additional template, and there's also the possibly pending inclusion of BIGTOP-732, where the regionserver (and possibly master) will require special treatment.

        I don't see any good reason for hbase.sh and hbase.sh.suse to be so different or for DEB packages to be even more different - we'll just need to test carefully to make sure we catch any legitimate differences. So with the exception of BIGTOP-732 requiring the regionserver to follow a very different structure, I think we ought to move toward cleaning up these templates - I'd be happy to help with the work once that's resolved.

        Show
        Sean Mackrory added a comment - I think using .svc files is a good idea - but as you pointed out, there are differences between hbase.sh and hbase.sh.suse. DEB packages are using an additional template, and there's also the possibly pending inclusion of BIGTOP-732 , where the regionserver (and possibly master) will require special treatment. I don't see any good reason for hbase.sh and hbase.sh.suse to be so different or for DEB packages to be even more different - we'll just need to test carefully to make sure we catch any legitimate differences. So with the exception of BIGTOP-732 requiring the regionserver to follow a very different structure, I think we ought to move toward cleaning up these templates - I'd be happy to help with the work once that's resolved.
        Hide
        Plamen Jeliazkov added a comment -

        Marking patch available.

        Show
        Plamen Jeliazkov added a comment - Marking patch available.
        Hide
        Plamen Jeliazkov added a comment -

        This should be the real fix now. CHKCONFIG is set to different start and stop priority levels for regionserver and master. SUSE differences are preserved through the hbase.sh.suse file.

        I have ran an HBase RPM build and verified the init script has the proper CHKCONFIG settings.

        This patch should be ready for commit. Peter/Cos/Bruno, please review.

        Show
        Plamen Jeliazkov added a comment - This should be the real fix now. CHKCONFIG is set to different start and stop priority levels for regionserver and master. SUSE differences are preserved through the hbase.sh.suse file. I have ran an HBase RPM build and verified the init script has the proper CHKCONFIG settings. This patch should be ready for commit. Peter/Cos/Bruno, please review.
        Hide
        Plamen Jeliazkov added a comment -

        Sorry actually I think the entire logic of my patch needs to be re-done. I see that hbase.sh and hbase.sh.suse have different settings – the .svc file route would have tried to merged them. I think it is best to keep them separate after all and handle just setting the CHKCONFIG.

        Show
        Plamen Jeliazkov added a comment - Sorry actually I think the entire logic of my patch needs to be re-done. I see that hbase.sh and hbase.sh.suse have different settings – the .svc file route would have tried to merged them. I think it is best to keep them separate after all and handle just setting the CHKCONFIG.
        Hide
        Plamen Jeliazkov added a comment -

        I think my patch still needs an update. The .scv files need all the properties set, correct? Currently I only set CHKCONFIG; but I think DESC, DAEMON, etc, also need to be set.

        Show
        Plamen Jeliazkov added a comment - I think my patch still needs an update. The .scv files need all the properties set, correct? Currently I only set CHKCONFIG; but I think DESC, DAEMON, etc, also need to be set.
        Hide
        Plamen Jeliazkov added a comment -

        New patch. This runs and creates an RPM. Cannot verify that this fixes the issue though.

        Show
        Plamen Jeliazkov added a comment - New patch. This runs and creates an RPM. Cannot verify that this fixes the issue though.
        Hide
        Plamen Jeliazkov added a comment -

        Attaching patch. Running "make hbase-rpm" with this patch seems to work; can't verify that CHKCONFIG is being overriden by .svc files though.

        Show
        Plamen Jeliazkov added a comment - Attaching patch. Running "make hbase-rpm" with this patch seems to work; can't verify that CHKCONFIG is being overriden by .svc files though.

          People

          • Assignee:
            Plamen Jeliazkov
            Reporter:
            Peter Linnell
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development