Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4631

Support customized call back method during failover automatically.

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: ha

      Description

      ZKFC add HealthCallbacks by default, this can do quiteElection at least. but we often want to be alerted if there is fail over occurring(such as send email, short messages), especially for prod cluster.

      There is a configured fence script. maybe we can put all these logics in the script. but in reasonable, fence script does only one thing: fence

      So I added this patch, we can configure customized HM callback method, if there is no configuration, then only HealthCallbacks is added.

        Activity

        Fengdong Yu created issue -
        Fengdong Yu made changes -
        Field Original Value New Value
        Attachment HDFS-4631.patch [ 12575297 ]
        Fengdong Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Labels features ha hadoop
        Fengdong Yu made changes -
        Description ZKFC add HealthCallbacks bu default, this can do quiteElection at least. but we often want to be alerted if there is fail over occurring(such as send email, short message), especially for prod cluster.

        There is a configured fence script. maybe we can put all these logics in the script. but in reasonable, fence script does only one thing: fence :)

        So I added this patch, we can configure customized HM callback method, if there is no configration, then only HealthCallbacks is added.
        ZKFC add HealthCallbacks by default, this can do quiteElection at least. but we often want to be alerted if there is fail over occurring(such as send email, short messages), especially for prod cluster.

        There is a configured fence script. maybe we can put all these logics in the script. but in reasonable, fence script does only one thing: fence :)

        So I added this patch, we can configure customized HM callback method, if there is no configuration, then only HealthCallbacks is added.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12575297/HDFS-4631.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4138//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575297/HDFS-4631.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4138//console This message is automatically generated.
        Fengdong Yu made changes -
        Attachment HDFS-4631.patch [ 12575297 ]
        Fengdong Yu made changes -
        Attachment HDFS-4631.patch [ 12575300 ]
        Fengdong Yu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Fengdong Yu made changes -
        Attachment HDFS-4631.patch [ 12575300 ]
        Fengdong Yu made changes -
        Attachment HDFS-4631.patch [ 12575301 ]
        Fengdong Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4139//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4139//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4139//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4139//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Nice idea. A couple thoughts on the patch:

        • rather than just instantiating one, we should allow a list using conf.getInstances().
        • this doesn't seem to implement exactly what you've described here – it won't tell you whether or not a failover has triggered, only when the health status changes. If your only goal is to monitor the health state, you could just poll "haamin -checkHealth" yourself in a loop, or instantiate HealthMonitor from your own Java program. It sounded to me like you want to know when it actually has initiated a failover?
        Show
        Todd Lipcon added a comment - Nice idea. A couple thoughts on the patch: rather than just instantiating one, we should allow a list using conf.getInstances(). this doesn't seem to implement exactly what you've described here – it won't tell you whether or not a failover has triggered, only when the health status changes. If your only goal is to monitor the health state, you could just poll "haamin -checkHealth" yourself in a loop, or instantiate HealthMonitor from your own Java program. It sounded to me like you want to know when it actually has initiated a failover?
        Hide
        Fengdong Yu added a comment -

        Yes Todd.
        I just want to be informed when start to do failover.

        I will try hdfs haadmin -checkhealt in a loop

        Show
        Fengdong Yu added a comment - Yes Todd. I just want to be informed when start to do failover. I will try hdfs haadmin -checkhealt in a loop
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Fengdong Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Fengdong Yu added a comment -

        Hi,

        I do think we need this patch for customized callback during state changed.(maybe do failover during state change, maybe not)

        but in the prod cluster, we still need get informed when there is health state changed.

        Show
        Fengdong Yu added a comment - Hi, I do think we need this patch for customized callback during state changed.(maybe do failover during state change, maybe not) but in the prod cluster, we still need get informed when there is health state changed.
        Allen Wittenauer made changes -
        Fix Version/s 3.0.0 [ 12320356 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch
        against trunk revision 14e2639.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8020//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch against trunk revision 14e2639. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8020//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10660/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10660/console This message was automatically generated.
        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10668/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12575301/HDFS-4631.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10668/console This message was automatically generated.
        Allen Wittenauer made changes -
        Labels features ha hadoop BB2015-05-TBR features ha hadoop
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        11d 9h 49m 2 Todd Lipcon 05/Apr/13 21:01
        Open Open Patch Available Patch Available
        5h 2m 3 Fengdong Yu 06/Apr/13 01:58

          People

          • Assignee:
            Unassigned
            Reporter:
            Fengdong Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 0.5m
              0.5m
              Remaining:
              Remaining Estimate - 0.5m
              0.5m
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development