Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3918

EditLogTailer shouldn't log WARN when other node is in standby mode

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: ha
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If both nodes are in standby mode, each will be trying to roll the others' logs, which results in errors like:

      Unable to trigger a roll of the active NN
      org.apache.hadoop.ipc.StandbyException: Operation category JOURNAL is not supported in state standby

      We should catch this specific exception and not log it at WARN level, since it's expected behavior.

      1. hdfs-3918.txt
        2 kB
        Todd Lipcon

        Activity

        Hide
        tlipcon Todd Lipcon added a comment -

        Attached patch cleans up the logs.

        I now see the following:

        12/09/20 19:30:36 INFO ha.EditLogTailer: Triggering log roll on remote NameNode todd-w510/127.0.0.1:8022
        12/09/20 19:30:36 INFO ha.EditLogTailer: Skipping log roll. Remote node is not in Active state: Operation category JOURNAL is not supported in state standby

        Show
        tlipcon Todd Lipcon added a comment - Attached patch cleans up the logs. I now see the following: 12/09/20 19:30:36 INFO ha.EditLogTailer: Triggering log roll on remote NameNode todd-w510/127.0.0.1:8022 12/09/20 19:30:36 INFO ha.EditLogTailer: Skipping log roll. Remote node is not in Active state: Operation category JOURNAL is not supported in state standby
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12545992/hdfs-3918.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDatanodeBlockScanner

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3222//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3222//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545992/hdfs-3918.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3222//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3222//console This message is automatically generated.
        Hide
        qwertymaniac Harsh J added a comment -

        Change looks good, but just so that the string change not cause an issue later (since we are apparently splitting it), can we have some form of regression-catch to ensure [0] is always what we want it to be?

        Not very necessary, but would be good to have. If its tough to do, we can punt. +1 with this addressed (in reply or in patch).

        Show
        qwertymaniac Harsh J added a comment - Change looks good, but just so that the string change not cause an issue later (since we are apparently splitting it), can we have some form of regression-catch to ensure [0] is always what we want it to be? Not very necessary, but would be good to have. If its tough to do, we can punt. +1 with this addressed (in reply or in patch).
        Hide
        tlipcon Todd Lipcon added a comment -

        Hmm, not sure exactly what kind of test you mean. I'm generally assuming that the exception response's first line is some kind of readable error message, which shouldn't change in the future.

        Show
        tlipcon Todd Lipcon added a comment - Hmm, not sure exactly what kind of test you mean. I'm generally assuming that the exception response's first line is some kind of readable error message, which shouldn't change in the future.
        Hide
        qwertymaniac Harsh J added a comment -

        Thanks for looking into that comment; yes its fine to go as-is too, +1.

        Show
        qwertymaniac Harsh J added a comment - Thanks for looking into that comment; yes its fine to go as-is too, +1.
        Hide
        qwertymaniac Harsh J added a comment -

        Sorry on delay here - I recalled this after having read a NN log for troubleshooting earlier today. Verified the improvement (and the patch) still applies cleanly, and committed to branch-2 and trunk. Thanks again Todd!

        Show
        qwertymaniac Harsh J added a comment - Sorry on delay here - I recalled this after having read a NN log for troubleshooting earlier today. Verified the improvement (and the patch) still applies cleanly, and committed to branch-2 and trunk. Thanks again Todd!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7472 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7472/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7472 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7472/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/149/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/149/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #883 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/883/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #883 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/883/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2081 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2081/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2081 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2081/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/140/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/140/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/149/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/149/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2099 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2099/)
        HDFS-3918. EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2099 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2099/ ) HDFS-3918 . EditLogTailer shouldn't log WARN when other node is in standby mode. Contributed by Todd Lipcon. (harsh: rev cce66ba3c9ec293e8ba1afd0eb518c7ca0bbc7c9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            tlipcon Todd Lipcon
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development