Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9332

Fix Precondition failures from NameNodeEditLogRoller while saving namespace

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      The check for the # of txns in the open edit log does not first check that an edit log segment is open, leading to a Precondition failure. This surfaced at HDFS-7871 which fixed that it was printing in a tight loop, but the cause of the Precondition failure is still present.

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Simple patch attached. FSNamesystem already has a method which does the correct validation, so just call that.

        Show
        andrew.wang Andrew Wang added a comment - Simple patch attached. FSNamesystem already has a method which does the correct validation, so just call that.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Annie Wang for reporting the issue and patch, +1 pending jenkins.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Annie Wang for reporting the issue and patch, +1 pending jenkins.
        Hide
        liuml07 Mingliang Liu added a comment -

        Looks good to me. +1 (non-bing)

        Show
        liuml07 Mingliang Liu added a comment - Looks good to me. +1 (non-bing)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 7s docker + precommit patch detected.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 3m 11s trunk passed
        +1 compile 0m 42s trunk passed with JDK v1.8.0_60
        +1 compile 0m 39s trunk passed with JDK v1.7.0_79
        +1 checkstyle 0m 18s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs
        +1 javadoc 1m 19s trunk passed with JDK v1.8.0_60
        +1 javadoc 2m 8s trunk passed with JDK v1.7.0_79
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.8.0_60
        +1 javac 0m 38s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.7.0_79
        +1 javac 0m 36s the patch passed
        -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 249, now 249).
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 18s the patch passed
        +1 javadoc 1m 16s the patch passed with JDK v1.8.0_60
        +1 javadoc 2m 4s the patch passed with JDK v1.7.0_79
        -1 unit 65m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_60.
        -1 unit 60m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_79.
        -1 asflicense 0m 23s Patch generated 56 ASF License warnings.
        148m 25s



        Reason Tests
        JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage
          hadoop.fs.viewfs.TestViewFsAtHdfsRoot
          hadoop.hdfs.TestBlockReaderLocal
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000



        Subsystem Report/Notes
        Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769421/HDFS-9332.001.patch
        JIRA Issue HDFS-9332
        Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile
        uname Linux bbbd52f7ef27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh
        git revision trunk / 65f53f2
        Default Java 1.7.0_79
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt
        JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13260/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 228MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13260/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 3m 11s trunk passed +1 compile 0m 42s trunk passed with JDK v1.8.0_60 +1 compile 0m 39s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 18s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 2m 13s hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs +1 javadoc 1m 19s trunk passed with JDK v1.8.0_60 +1 javadoc 2m 8s trunk passed with JDK v1.7.0_79 +1 mvninstall 0m 45s the patch passed +1 compile 0m 38s the patch passed with JDK v1.8.0_60 +1 javac 0m 38s the patch passed +1 compile 0m 36s the patch passed with JDK v1.7.0_79 +1 javac 0m 36s the patch passed -1 checkstyle 0m 17s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 249, now 249). +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 18s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_60 +1 javadoc 2m 4s the patch passed with JDK v1.7.0_79 -1 unit 65m 17s hadoop-hdfs in the patch failed with JDK v1.8.0_60. -1 unit 60m 46s hadoop-hdfs in the patch failed with JDK v1.7.0_79. -1 asflicense 0m 23s Patch generated 56 ASF License warnings. 148m 25s Reason Tests JDK v1.7.0_79 Failed junit tests hadoop.hdfs.TestDFSUpgradeFromImage   hadoop.fs.viewfs.TestViewFsAtHdfsRoot   hadoop.hdfs.TestBlockReaderLocal   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-10-28 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12769421/HDFS-9332.001.patch JIRA Issue HDFS-9332 Optional Tests asflicense javac javadoc mvninstall unit findbugs checkstyle compile uname Linux bbbd52f7ef27 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/apache-yetus-67f42f1/precommit/personality/hadoop.sh git revision trunk / 65f53f2 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13260/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13260/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13260/console This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Failed tests are unrelated, ran them locally. I also filed YETUS-146 for what look like some Yetus display bugs. Will commit shortly.

        Show
        andrew.wang Andrew Wang added a comment - Failed tests are unrelated, ran them locally. I also filed YETUS-146 for what look like some Yetus display bugs. Will commit shortly.
        Hide
        andrew.wang Andrew Wang added a comment -

        Fixed, thanks for reviewing Yongjun and Mingliang!

        Show
        andrew.wang Andrew Wang added a comment - Fixed, thanks for reviewing Yongjun and Mingliang!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8725/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8725 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8725/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #602 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/602/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2544 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2544/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1337 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1337/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #614 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/614/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #552 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/552/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #552 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/552/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2490/)
        HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2490 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2490/ ) HDFS-9332 . Fix Precondition failures from NameNodeEditLogRoller while (wang: rev 888c6245e20ba6bdaa57d16b5c62b4a9eda2cdaf) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development