Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9555

LazyPersistFileScrubber should still sleep if there are errors in the clear progress

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None

      Description

      If LazyPersistFileScrubber.clearCorruptLazyPersistFiles throw an exception in run(), there will be no sleep logic so it will restart immediately. However it may be still fail so there are too many ERROR logs in namenode said "Ignoring exception in LazyPersistFileScrubber".

      We need sleep if we catch the exception.

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 3s trunk passed
        +1 compile 0m 43s trunk passed with JDK v1.8.0_66
        +1 compile 0m 43s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 2s trunk passed
        +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.8.0_66
        +1 javac 0m 42s the patch passed
        +1 compile 0m 45s the patch passed with JDK v1.7.0_91
        +1 javac 0m 45s the patch passed
        -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 193, now 193).
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 9s the patch passed
        +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91
        -1 unit 55m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 60m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 19s Patch generated 56 ASF License warnings.
        144m 26s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner
          hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
          hadoop.hdfs.server.namenode.ha.TestEditLogTailer
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
        JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFileTruncate



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777436/9555-v1.patch
        JIRA Issue HDFS-9555
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d804dc7fe779 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7fb212e
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13857/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus 0.1.0 http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13857/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 3s trunk passed +1 compile 0m 43s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 52s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 51s the patch passed +1 compile 0m 42s the patch passed with JDK v1.8.0_66 +1 javac 0m 42s the patch passed +1 compile 0m 45s the patch passed with JDK v1.7.0_91 +1 javac 0m 45s the patch passed -1 checkstyle 0m 16s Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 193, now 193). +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 9s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_91 -1 unit 55m 50s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 60m 39s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 19s Patch generated 56 ASF License warnings. 144m 26s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.server.namenode.ha.TestEditLogTailer JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777436/9555-v1.patch JIRA Issue HDFS-9555 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d804dc7fe779 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fb212e findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13857/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13857/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13857/console This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Thanks for reporting this.

        I'm not sure swallowing Exception instead of IOException is a good idea after clearCorruptLazyPersistFiles, but skipping the sleep logic makes little sense in the current code.

        +1 (non-binding).

        Show
        liuml07 Mingliang Liu added a comment - Thanks for reporting this. I'm not sure swallowing Exception instead of IOException is a good idea after clearCorruptLazyPersistFiles , but skipping the sleep logic makes little sense in the current code. +1 (non-binding).
        Hide
        djp Junping Du added a comment -

        Move all non-critical pending issues out of 2.6.4 into 2.6.5.

        Show
        djp Junping Du added a comment - Move all non-critical pending issues out of 2.6.4 into 2.6.5.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 16s trunk passed
        +1 compile 0m 46s trunk passed with JDK v1.8.0_74
        +1 compile 0m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 2m 9s trunk passed
        +1 javadoc 1m 14s trunk passed with JDK v1.8.0_74
        +1 javadoc 1m 51s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.8.0_74
        +1 javac 0m 41s the patch passed
        +1 compile 0m 41s the patch passed with JDK v1.7.0_95
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 20s the patch passed
        +1 javadoc 1m 10s the patch passed with JDK v1.8.0_74
        +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95
        -1 unit 60m 33s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
        -1 unit 55m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        143m 14s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.shortcircuit.TestShortCircuitCache
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
          hadoop.hdfs.TestHFlush
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777436/9555-v1.patch
        JIRA Issue HDFS-9555
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fd504c5de1de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ed1e23f
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14879/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14879/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_74 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 51s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 50s the patch passed +1 compile 0m 41s the patch passed with JDK v1.8.0_74 +1 javac 0m 41s the patch passed +1 compile 0m 41s the patch passed with JDK v1.7.0_95 +1 javac 0m 41s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 20s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_74 +1 javadoc 1m 53s the patch passed with JDK v1.7.0_95 -1 unit 60m 33s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 55m 18s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 143m 14s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.shortcircuit.TestShortCircuitCache JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.ha.TestBootstrapStandby   hadoop.hdfs.TestHFlush   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12777436/9555-v1.patch JIRA Issue HDFS-9555 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fd504c5de1de 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed1e23f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14879/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14879/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14879/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        +1. ltgm
        Phil Yang: do you mind creating a patch for branch-2.7 since the current patch doesn't apply cleanly to branch-2.7 ?
        Kihwal Lee: can you please review and commit if it looks good to you.

        Show
        shahrs87 Rushabh S Shah added a comment - +1. ltgm Phil Yang : do you mind creating a patch for branch-2.7 since the current patch doesn't apply cleanly to branch-2.7 ? Kihwal Lee : can you please review and commit if it looks good to you.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        +1 (non-binding)

        Show
        shahrs87 Rushabh S Shah added a comment - +1 (non-binding)
        Hide
        kihwal Kihwal Lee added a comment -

        +1 the patch looks good.

        Show
        kihwal Kihwal Lee added a comment - +1 the patch looks good.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9654 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9654/)
        HDFS-9555. LazyPersistFileScrubber should still sleep if there are (kihwal: rev 5d4255a80156d2cacfea8184b41805070223d3a9)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9654 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9654/ ) HDFS-9555 . LazyPersistFileScrubber should still sleep if there are (kihwal: rev 5d4255a80156d2cacfea8184b41805070223d3a9) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        Hide
        kihwal Kihwal Lee added a comment -

        I've committed this to trunk through branch-2.7. Thanks for the patch, Phil Yang.

        Show
        kihwal Kihwal Lee added a comment - I've committed this to trunk through branch-2.7. Thanks for the patch, Phil Yang .
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            yangzhe1991 Phil Yang
            Reporter:
            yangzhe1991 Phil Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development