Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7871

NameNodeEditLogRoller can keep printing "Swallowing exception" message

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In one of our customer's cluster, we saw the NameNode kept printing out the following error msg and thus got stuck for several minutes:

      ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Swallowing exception in NameNodeEditLogRoller:
      java.lang.IllegalStateException: Bad state: BETWEEN_LOG_SEGMENTS
              at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLog.getCurSegmentTxId(FSEditLog.java:493)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4358)
              at java.lang.Thread.run(Thread.java:745)
      

        Issue Links

          Activity

          Hide
          jingzhao Jing Zhao added a comment -

          Looking at the code, since FSEditLog#getCurSegmentTxId can throw an exception, Thread.sleep(sleepIntervalMs) should be moved out of the try-catch block.

          Show
          jingzhao Jing Zhao added a comment - Looking at the code, since FSEditLog#getCurSegmentTxId can throw an exception, Thread.sleep(sleepIntervalMs) should be moved out of the try-catch block.
          Hide
          jingzhao Jing Zhao added a comment -

          Patch to move the sleep expression out.

          Show
          jingzhao Jing Zhao added a comment - Patch to move the sleep expression out.
          Hide
          hitliuyi Yi Liu added a comment -

          +1, make sense.

          Show
          hitliuyi Yi Liu added a comment - +1, make sense.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702005/HDFS-7871.000.patch
          against trunk revision 5d0bae5.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9702//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9702//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702005/HDFS-7871.000.patch against trunk revision 5d0bae5. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9702//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9702//console This message is automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for the review, Yi Liu!

          The failed tests should be unrelated, and they all passed in my local machine. I've committed this to trunk and branch-2.

          Show
          jingzhao Jing Zhao added a comment - Thanks for the review, Yi Liu ! The failed tests should be unrelated, and they all passed in my local machine. I've committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7241 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7241/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7241 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7241/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #121 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/121/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #121 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/121/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #855 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/855/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #855 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/855/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2053/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2053/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/112/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/112/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/121/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/121/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2071 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2071/)
          HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2071 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2071/ ) HDFS-7871 . NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. (jing9: rev b442aeec95abfa1c6f835a116dfe6e186b0d841d) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cnauroth Chris Nauroth added a comment -

          Nice find, Jing. Thanks for the patch!

          Show
          cnauroth Chris Nauroth added a comment - Nice find, Jing. Thanks for the patch!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Sangjin Lee backported this to 2.6.1, the patch applies cleanly. I just pushed the commit to 2.6.1 after running compilation.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Sangjin Lee backported this to 2.6.1, the patch applies cleanly. I just pushed the commit to 2.6.1 after running compilation.

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              jingzhao Jing Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development