Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11087

NamenodeFsck should check if the output writer is still writable.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 2.8.0, 2.7.4
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      NamenodeFsck keeps running even after the client was interrupted. So if you start fsck / on a large namespace and kill the client, the NameNode will keep traversing the tree for hours although there is nobody to receive the result.

      1. HDFS-11087.branch-2.000.patch
        0.9 kB
        Erik Krogen
      2. HDFS-11087-branch-2.000.patch
        0.9 kB
        Erik Krogen
      3. HDFS-11087-branch-2.001.patch
        1 kB
        Erik Krogen

        Activity

        Hide
        shv Konstantin Shvachko added a comment -

        The problem is that PrintWriter.ensureOpen() does not really check the status of the underlying Writer to the HttpConnection, which has already been closed.
        This is probably Jetty 6 problem. On trunk with upgraded Jetty I see that server side fsck fails after client is interrupted. So this is applicable to all v.2 branches only.
        I propose to fix this by calling PrintWriter.checkError() before each round of printing in NamenodeFsck.check().

        Show
        shv Konstantin Shvachko added a comment - The problem is that PrintWriter.ensureOpen() does not really check the status of the underlying Writer to the HttpConnection, which has already been closed. This is probably Jetty 6 problem. On trunk with upgraded Jetty I see that server side fsck fails after client is interrupted. So this is applicable to all v.2 branches only. I propose to fix this by calling PrintWriter.checkError() before each round of printing in NamenodeFsck.check() .
        Hide
        xkrogen Erik Krogen added a comment -

        Attaching v000 patch.

        Since checkError internally calls flush, probably best not to do it too frequently. We already flush after processing every 100 files, so I piggybacked off of that, replacing the call to flush with a call to checkError which will then throw an exception if the stream has been closed / otherwise failed.

        Show
        xkrogen Erik Krogen added a comment - Attaching v000 patch. Since checkError internally calls flush , probably best not to do it too frequently. We already flush after processing every 100 files, so I piggybacked off of that, replacing the call to flush with a call to checkError which will then throw an exception if the stream has been closed / otherwise failed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 5m 34s Docker failed to build yetus/hadoop:b59b8b7.



        Subsystem Report/Notes
        JIRA Issue HDFS-11087
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838247/HDFS-11087.branch-2.000.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17489/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 5m 34s Docker failed to build yetus/hadoop:b59b8b7. Subsystem Report/Notes JIRA Issue HDFS-11087 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838247/HDFS-11087.branch-2.000.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17489/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        HDFS-11087.branch-2.000.patch -> HDFS-11087-branch-2.000.patch

        Show
        liuml07 Mingliang Liu added a comment - HDFS-11087 .branch-2.000.patch -> HDFS-11087 -branch-2.000.patch
        Hide
        xkrogen Erik Krogen added a comment -

        Whoops, thank you!

        Show
        xkrogen Erik Krogen added a comment - Whoops, thank you!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 41s branch-2 passed
        +1 compile 0m 39s branch-2 passed with JDK v1.8.0_111
        +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111
        +1 checkstyle 0m 28s branch-2 passed
        +1 mvnsite 0m 51s branch-2 passed
        +1 mvneclipse 0m 15s branch-2 passed
        +1 findbugs 1m 56s branch-2 passed
        +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_111
        +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_111
        +1 mvninstall 0m 43s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_111
        +1 javac 0m 36s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.7.0_111
        +1 javac 0m 40s the patch passed
        +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 105 unchanged - 2 fixed = 105 total (was 107)
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 6s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_111
        +1 javadoc 1m 32s the patch passed with JDK v1.7.0_111
        -1 unit 48m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        122m 37s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.hdfs.tools.TestDFSAdmin
        JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-11087
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838251/HDFS-11087-branch-2.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 6657d7c280bd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 16430c6
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17491/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17491/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 41s branch-2 passed +1 compile 0m 39s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 43s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 28s branch-2 passed +1 mvnsite 0m 51s branch-2 passed +1 mvneclipse 0m 15s branch-2 passed +1 findbugs 1m 56s branch-2 passed +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_111 +1 javadoc 1m 35s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 43s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_111 +1 javac 0m 36s the patch passed +1 compile 0m 40s the patch passed with JDK v1.7.0_111 +1 javac 0m 40s the patch passed +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 105 unchanged - 2 fixed = 105 total (was 107) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_111 +1 javadoc 1m 32s the patch passed with JDK v1.7.0_111 -1 unit 48m 16s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 122m 37s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.hdfs.tools.TestDFSAdmin JDK v1.7.0_111 Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11087 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838251/HDFS-11087-branch-2.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6657d7c280bd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 16430c6 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17491/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17491/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17491/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment - - edited

        This makes sense, Erik.
        Could you please add a comment before checkError() based on its JavaDoc.
        It is not clear that the intent here to flush the stream while checking the error.
        +1 on the patch modular the comment line. Will commit once you add it.

        Show
        shv Konstantin Shvachko added a comment - - edited This makes sense, Erik. Could you please add a comment before checkError() based on its JavaDoc. It is not clear that the intent here to flush the stream while checking the error. +1 on the patch modular the comment line. Will commit once you add it.
        Hide
        xkrogen Erik Krogen added a comment -

        Thanks for the review, Konstantin Shvachko. Attaching v001 patch with an additional comment.

        Show
        xkrogen Erik Krogen added a comment - Thanks for the review, Konstantin Shvachko . Attaching v001 patch with an additional comment.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 49s branch-2 passed
        +1 compile 0m 48s branch-2 passed with JDK v1.8.0_101
        +1 compile 0m 46s branch-2 passed with JDK v1.7.0_111
        +1 checkstyle 0m 27s branch-2 passed
        +1 mvnsite 0m 54s branch-2 passed
        +1 mvneclipse 0m 16s branch-2 passed
        +1 findbugs 2m 3s branch-2 passed
        +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_101
        +1 javadoc 1m 43s branch-2 passed with JDK v1.7.0_111
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 46s the patch passed with JDK v1.8.0_101
        +1 javac 0m 46s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.7.0_111
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 105 unchanged - 2 fixed = 105 total (was 107)
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 12s the patch passed
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_101
        +1 javadoc 1m 36s the patch passed with JDK v1.7.0_111
        -1 unit 68m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        164m 45s



        Reason Tests
        JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency
          hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot
          hadoop.tools.TestDelegationTokenFetcher
        JDK v1.7.0_111 Failed junit tests hadoop.tools.TestDelegationTokenFetcher



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-11087
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838638/HDFS-11087-branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 30d1308aab4f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 9b0a2cb
        Default Java 1.7.0_111
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17537/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
        JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17537/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17537/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 49s branch-2 passed +1 compile 0m 48s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 46s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 27s branch-2 passed +1 mvnsite 0m 54s branch-2 passed +1 mvneclipse 0m 16s branch-2 passed +1 findbugs 2m 3s branch-2 passed +1 javadoc 0m 55s branch-2 passed with JDK v1.8.0_101 +1 javadoc 1m 43s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 46s the patch passed +1 compile 0m 46s the patch passed with JDK v1.8.0_101 +1 javac 0m 46s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_111 +1 javac 0m 42s the patch passed +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 105 unchanged - 2 fixed = 105 total (was 107) +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 36s the patch passed with JDK v1.7.0_111 -1 unit 68m 45s hadoop-hdfs in the patch failed with JDK v1.7.0_111. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 164m 45s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency   hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot   hadoop.tools.TestDelegationTokenFetcher JDK v1.7.0_111 Failed junit tests hadoop.tools.TestDelegationTokenFetcher Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11087 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838638/HDFS-11087-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 30d1308aab4f 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 9b0a2cb Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17537/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17537/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17537/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        I just committed this to branches 2, 2.8, 2.7, but not to trunk as it is not a problem there as noted earlier.
        Thank you Erik Krogen.

        Show
        shv Konstantin Shvachko added a comment - I just committed this to branches 2, 2.8, 2.7, but not to trunk as it is not a problem there as noted earlier. Thank you Erik Krogen .
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Looks to be CHANGES.txt not updated for this. can you update..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Looks to be CHANGES.txt not updated for this. can you update..?
        Hide
        shv Konstantin Shvachko added a comment -

        Hey Brahma Reddy Battula, looks like everybody is forgetting to update CHANGES.txt for branch-2.7. Should we consider adding a pre-commit hook?

        Show
        shv Konstantin Shvachko added a comment - Hey Brahma Reddy Battula , looks like everybody is forgetting to update CHANGES.txt for branch-2.7. Should we consider adding a pre-commit hook?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        That will be good idea .Everybody will aware of it before committing to branch-2.7 and we might not miss any entry.

        Should we consider adding a pre-commit hook?

        can we do this..?

        Even I sent mail in common-dev (sub: Updation of Change.txt in branch-2.7 on Sept-30),but I did not hear any thoughts there.
        If we need any discussion in mailing list ,may be we can use this..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - That will be good idea .Everybody will aware of it before committing to branch-2.7 and we might not miss any entry. Should we consider adding a pre-commit hook? can we do this..? Even I sent mail in common-dev (sub: Updation of Change.txt in branch-2.7 on Sept-30),but I did not hear any thoughts there. If we need any discussion in mailing list ,may be we can use this..?

          People

          • Assignee:
            xkrogen Erik Krogen
            Reporter:
            shv Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development