Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11896

Non-dfsUsed will be doubled on dead node re-registration

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Scenario:
      i)Make you sure you've non-dfs data.
      ii) Stop Datanode
      iii) wait it becomes dead
      iv) now restart and check the non-dfs data

      1. HDFS-11896-branch-2.7-001.patch
        7 kB
        Brahma Reddy Battula
      2. HDFS-11896-branch-2.7-002.patch
        8 kB
        Brahma Reddy Battula
      3. HDFS-11896.patch
        5 kB
        Brahma Reddy Battula
      4. HDFS-11896-002.patch
        5 kB
        Brahma Reddy Battula
      5. HDFS-11896-003.patch
        7 kB
        Brahma Reddy Battula
      6. HDFS-11896-branch-2.7-003.patch
        10 kB
        Brahma Reddy Battula
      7. HDFS-11896-004.patch
        7 kB
        Brahma Reddy Battula
      8. HDFS-11896-005.patch
        7 kB
        Brahma Reddy Battula
      9. HDFS-11896-branch-2.7-004.patch
        9 kB
        Brahma Reddy Battula
      10. HDFS-11896-006.patch
        7 kB
        Brahma Reddy Battula
      11. HDFS-11896-007.patch
        7 kB
        Brahma Reddy Battula
      12. HDFS-11896-branch-2.7-005.patch
        9 kB
        Brahma Reddy Battula
      13. HDFS-11896-branch-2.7-006.patch
        8 kB
        Zhe Zhang
      14. HDFS-11896-branch-2.7-008.patch
        8 kB
        Konstantin Shvachko
      15. HDFS-11896-008.patch
        7 kB
        Konstantin Shvachko

        Activity

        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        This will not happen after HDFS-9034 since stats will be added after updateheartbeat.

        Uploaded the patch to fix in branch-2.7.Kindly review..

        Even we can backport HDFS-9034 to branch-2.7..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - This will not happen after HDFS-9034 since stats will be added after updateheartbeat . Uploaded the patch to fix in branch-2.7.Kindly review.. Even we can backport HDFS-9034 to branch-2.7..?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 3m 3s Docker failed to build yetus/hadoop:c420dfe.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19647/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 3m 3s Docker failed to build yetus/hadoop:c420dfe. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19647/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 8m 36s Docker failed to build yetus/hadoop:c420dfe.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19649/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 8m 36s Docker failed to build yetus/hadoop:c420dfe. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19649/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 15s Docker failed to build yetus/hadoop:c420dfe.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19650/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 15s Docker failed to build yetus/hadoop:c420dfe. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19650/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        This will not happen after HDFS-9034 since stats will be added after updateheartbeat.

        Can we do just that in this jira for 2.7. Will it fix NonDfsUsed reporting issue?

        Brahma Reddy Battula, I think your patch is useful in all versions, should not be restricted to 2.7 only, especially the test case. Why don't you target this change for trunk, etc.
        For 2.7 we should be fixing the NonDfsUsed reporting only. I believe something is not resetting during DN registration, which it does in other versions.

        Show
        shv Konstantin Shvachko added a comment - This will not happen after HDFS-9034 since stats will be added after updateheartbeat. Can we do just that in this jira for 2.7. Will it fix NonDfsUsed reporting issue? Brahma Reddy Battula , I think your patch is useful in all versions, should not be restricted to 2.7 only, especially the test case. Why don't you target this change for trunk, etc. For 2.7 we should be fixing the NonDfsUsed reporting only. I believe something is not resetting during DN registration, which it does in other versions.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 5m 59s branch-2.7 passed
        +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 27s branch-2.7 passed
        +1 mvnsite 0m 57s branch-2.7 passed
        +1 mvneclipse 0m 14s branch-2.7 passed
        +1 findbugs 2m 51s branch-2.7 passed
        +1 javadoc 0m 57s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 47s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.8.0_131
        +1 javac 0m 55s the patch passed
        +1 compile 0m 57s the patch passed with JDK v1.7.0_131
        +1 javac 0m 57s the patch passed
        -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 321 unchanged - 1 fixed = 323 total (was 322)
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 1474 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 38s The patch 70 line(s) with tabs.
        +1 findbugs 3m 8s the patch passed
        +1 javadoc 0m 54s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 45s the patch passed with JDK v1.7.0_131
        -1 unit 55m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        137m 50s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 75cf9a2b6094 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / cafee11
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19655/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19655/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 5m 59s branch-2.7 passed +1 compile 1m 1s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 27s branch-2.7 passed +1 mvnsite 0m 57s branch-2.7 passed +1 mvneclipse 0m 14s branch-2.7 passed +1 findbugs 2m 51s branch-2.7 passed +1 javadoc 0m 57s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 47s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 52s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_131 +1 javac 0m 55s the patch passed +1 compile 0m 57s the patch passed with JDK v1.7.0_131 +1 javac 0m 57s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 321 unchanged - 1 fixed = 323 total (was 322) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 1474 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 38s The patch 70 line(s) with tabs. +1 findbugs 3m 8s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 45s the patch passed with JDK v1.7.0_131 -1 unit 55m 19s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 137m 50s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870200/HDFS-11896-branch-2.7-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 75cf9a2b6094 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / cafee11 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19655/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19655/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19655/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Can we do just that in this jira for 2.7. Will it fix NonDfsUsed reporting issue?

        Yes,we can backport HDFS-9034.

        I think your patch is useful in all versions, should not be restricted to 2.7 only, especially the test case. Why don't you target this change for trunk, etc.

        Yes, it can be committed to all versions.But issue will not occur in other versions because nondfs is reset during the register before adding back stats. and test will not fail without source changes .

        IMO,we can merge attached patch to all versions which will fix the nondfsused and we can raise seperate issue to backport HDFS-9034.

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Can we do just that in this jira for 2.7. Will it fix NonDfsUsed reporting issue? Yes,we can backport HDFS-9034 . I think your patch is useful in all versions, should not be restricted to 2.7 only, especially the test case. Why don't you target this change for trunk, etc. Yes, it can be committed to all versions.But issue will not occur in other versions because nondfs is reset during the register before adding back stats. and test will not fail without source changes . IMO,we can merge attached patch to all versions which will fix the nondfsused and we can raise seperate issue to backport HDFS-9034 .
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Tried Backport of HDFS-9034,it may be incomplete without HDFS-7390 + HDFS-9223.So I added,only necessary changes.
        If you want me to break out HDFS-9034, I can do.

        Uploading the branch-2.7 patch..Kindly review.

        will upload trunk patch,which is just reset of nondfsused.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Tried Backport of HDFS-9034 ,it may be incomplete without HDFS-7390 + HDFS-9223 .So I added,only necessary changes. If you want me to break out HDFS-9034 , I can do. Uploading the branch-2.7 patch..Kindly review. will upload trunk patch,which is just reset of nondfsused .
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the trunk patch.
        As I mentioned earlier,testcase will pass without resetting the non-dfsused

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the trunk patch. As I mentioned earlier,testcase will pass without resetting the non-dfsused
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 5m 55s branch-2.7 passed
        +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 28s branch-2.7 passed
        +1 mvnsite 0m 58s branch-2.7 passed
        +1 findbugs 3m 5s branch-2.7 passed
        +1 javadoc 1m 6s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 56s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 0m 57s the patch passed
        +1 compile 1m 1s the patch passed with JDK v1.8.0_131
        +1 javac 1m 1s the patch passed
        +1 compile 0m 58s the patch passed with JDK v1.7.0_131
        +1 javac 0m 58s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 0m 57s the patch passed
        -1 whitespace 0m 0s The patch has 1819 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 42s The patch 70 line(s) with tabs.
        +1 findbugs 3m 5s the patch passed
        +1 javadoc 1m 2s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 48s the patch passed with JDK v1.7.0_131
        -1 unit 43m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        117m 41s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem
          hadoop.hdfs.TestCrcCorruption
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873284/HDFS-11896-branch-2.7-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8cf5a5bd122c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / f36da00
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19927/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19927/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 5m 55s branch-2.7 passed +1 compile 0m 58s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 28s branch-2.7 passed +1 mvnsite 0m 58s branch-2.7 passed +1 findbugs 3m 5s branch-2.7 passed +1 javadoc 1m 6s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 56s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 57s the patch passed +1 compile 1m 1s the patch passed with JDK v1.8.0_131 +1 javac 1m 1s the patch passed +1 compile 0m 58s the patch passed with JDK v1.7.0_131 +1 javac 0m 58s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 0m 57s the patch passed -1 whitespace 0m 0s The patch has 1819 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 42s The patch 70 line(s) with tabs. +1 findbugs 3m 5s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 48s the patch passed with JDK v1.7.0_131 -1 unit 43m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 117m 41s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873284/HDFS-11896-branch-2.7-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8cf5a5bd122c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / f36da00 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19927/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19927/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19927/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 54s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 48s the patch passed
        +1 javac 0m 48s the patch passed
        +1 checkstyle 0m 35s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 56s the patch passed
        +1 javadoc 0m 41s the patch passed
        -1 unit 95m 23s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        122m 14s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestPread
          hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873286/HDFS-11896.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ffcf63661ef6 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 942ecbb
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19928/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19928/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19928/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 54s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 0s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 95m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 122m 14s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestPread   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873286/HDFS-11896.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ffcf63661ef6 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 942ecbb Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19928/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19928/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19928/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 17s branch-2.7 passed
        +1 compile 1m 6s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 29s branch-2.7 passed
        +1 mvnsite 1m 4s branch-2.7 passed
        +1 findbugs 3m 11s branch-2.7 passed
        +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 56s branch-2.7 passed with JDK v1.7.0_131
        +1 mvninstall 0m 57s the patch passed
        +1 compile 1m 8s the patch passed with JDK v1.8.0_131
        +1 javac 1m 8s the patch passed
        +1 compile 1m 10s the patch passed with JDK v1.7.0_131
        +1 javac 1m 10s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 1m 3s the patch passed
        -1 whitespace 0m 0s The patch has 1819 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 39s The patch 70 line(s) with tabs.
        +1 findbugs 3m 25s the patch passed
        +1 javadoc 0m 58s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 41s the patch passed with JDK v1.7.0_131
        -1 unit 54m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        144m 48s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.ha.TestDNFencing
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.namenode.TestDeadDatanode
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873284/HDFS-11896-branch-2.7-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f18b6411ee51 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / f36da00
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19926/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19926/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 17s branch-2.7 passed +1 compile 1m 6s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 4s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 29s branch-2.7 passed +1 mvnsite 1m 4s branch-2.7 passed +1 findbugs 3m 11s branch-2.7 passed +1 javadoc 1m 4s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 56s branch-2.7 passed with JDK v1.7.0_131 +1 mvninstall 0m 57s the patch passed +1 compile 1m 8s the patch passed with JDK v1.8.0_131 +1 javac 1m 8s the patch passed +1 compile 1m 10s the patch passed with JDK v1.7.0_131 +1 javac 1m 10s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 1m 3s the patch passed -1 whitespace 0m 0s The patch has 1819 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 39s The patch 70 line(s) with tabs. +1 findbugs 3m 25s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 41s the patch passed with JDK v1.7.0_131 -1 unit 54m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 144m 48s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.ha.TestDNFencing   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.8.0_131 Timed out junit tests org.apache.hadoop.hdfs.server.namenode.TestAddBlockRetry JDK v1.7.0_131 Failed junit tests hadoop.hdfs.server.blockmanagement.TestNodeCount   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.namenode.TestDeadDatanode   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873284/HDFS-11896-branch-2.7-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f18b6411ee51 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / f36da00 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19926/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19926/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19926/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the trunk patch to fix the testcase.. will update branch-2.7 testcase once approach is fine to you.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the trunk patch to fix the testcase.. will update branch-2.7 testcase once approach is fine to you.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 16m 25s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 8s trunk passed
        +1 findbugs 1m 56s trunk passed
        +1 javadoc 0m 47s trunk passed
        +1 mvninstall 1m 0s the patch passed
        +1 compile 1m 0s the patch passed
        +1 javac 1m 0s the patch passed
        +1 checkstyle 0m 39s the patch passed
        +1 mvnsite 1m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 4s the patch passed
        +1 javadoc 0m 39s the patch passed
        -1 unit 87m 29s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        118m 5s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.namenode.TestDeadDatanode



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873412/HDFS-11896-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3ec92f400232 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7582ded
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19937/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19937/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19937/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 16m 25s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 8s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 1m 0s the patch passed +1 compile 1m 0s the patch passed +1 javac 1m 0s the patch passed +1 checkstyle 0m 39s the patch passed +1 mvnsite 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 4s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 87m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 118m 5s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.namenode.TestDeadDatanode Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873412/HDFS-11896-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3ec92f400232 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7582ded Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/19937/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19937/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19937/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Arpit Agarwal/Vinayakumar B if you get chance, can you look in this issue.will dig and fix the test failure next patch,it's passing in my local linux also..

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Arpit Agarwal / Vinayakumar B if you get chance, can you look in this issue.will dig and fix the test failure next patch,it's passing in my local linux also..
        Hide
        shv Konstantin Shvachko added a comment - - edited

        Spent some time debugging this. Found two problems:

        1. When heartbeat expires and DatanodeManager does removeDatanode() it does not reset nonDfsUsed, see DatanodeDescriptor.resetBlocks(), and some other fields, which it should.
          This exists on trunk and other versions.
        2. During registration HeartbeatManager.register() first calls addDatanode(), which increments stats, then resets DN usage to zeros via updateHeartbeatState().
          This was fixed by HDFS-9034, but still remains in branch-2.7.

        For branch-2.7 combination of these two bugs makes nonDfsUsed counted twice on re-registration.

        Show
        shv Konstantin Shvachko added a comment - - edited Spent some time debugging this. Found two problems: When heartbeat expires and DatanodeManager does removeDatanode() it does not reset nonDfsUsed , see DatanodeDescriptor.resetBlocks() , and some other fields, which it should. This exists on trunk and other versions. During registration HeartbeatManager.register() first calls addDatanode() , which increments stats , then resets DN usage to zeros via updateHeartbeatState() . This was fixed by HDFS-9034 , but still remains in branch-2.7. For branch-2.7 combination of these two bugs makes nonDfsUsed counted twice on re-registration.
        Hide
        shv Konstantin Shvachko added a comment - - edited

        I was thinking how to fix this. Think jira should fix (1) starting from trunk down to 2.7.4.
        For (2) there was an easier fix for HDFS-9034, just to swap 2 lines in HeartbeatManager.register(), that is first do updateHeartbeatState(), then addDatanode(). The former will reset everything to 0 in DND, including nonDfsUsed and the latter will be adding 0 then. This would avoid breaking followup jiras. We should just add the swapping in branch-2.7 patch here.

        Show
        shv Konstantin Shvachko added a comment - - edited I was thinking how to fix this. Think jira should fix (1) starting from trunk down to 2.7.4. For (2) there was an easier fix for HDFS-9034 , just to swap 2 lines in HeartbeatManager.register() , that is first do updateHeartbeatState() , then addDatanode() . The former will reset everything to 0 in DND, including nonDfsUsed and the latter will be adding 0 then. This would avoid breaking followup jiras. We should just add the swapping in branch-2.7 patch here.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Konstantin Shvachko thanks a lot for review and debugging this..

        For branch-2.7 combination of these two bugs makes nonDfsUsed counted twice on re-registration.

        Uploaded patch is combination only.
        and for trunk also patch is available with just reseting the non-dfs used.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Konstantin Shvachko thanks a lot for review and debugging this.. For branch-2.7 combination of these two bugs makes nonDfsUsed counted twice on re-registration. Uploaded patch is combination only. and for trunk also patch is available with just reseting the non-dfs used.
        Hide
        shv Konstantin Shvachko added a comment - - edited

        Hey Brahma Reddy Battula,

        • in your trunk patch you correctly reset nonDfsUsed to 0, but not other fields. I was thinking we could factor out the part of updateHeartbeatState() that sets the "total" fields excluding the update times and use it in resetBlocks(). That way we know all fields are updated even if somebody adds new ones.
        • In the 2.7 patch you follow the HDFS-9034 path introducing updateDnStat(), which will cause test failures as HDFS-9034 did. I recommend instead to swap lines in HeartbeatManager.register() without modifying addDatanode().
          Like this:
            synchronized void register(final DatanodeDescriptor d) {
              if (!d.isAlive) {
          -      addDatanode(d);
                 d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
          +      addDatanode(d);
              }
            }
          
        Show
        shv Konstantin Shvachko added a comment - - edited Hey Brahma Reddy Battula , in your trunk patch you correctly reset nonDfsUsed to 0, but not other fields. I was thinking we could factor out the part of updateHeartbeatState() that sets the "total" fields excluding the update times and use it in resetBlocks() . That way we know all fields are updated even if somebody adds new ones. In the 2.7 patch you follow the HDFS-9034 path introducing updateDnStat() , which will cause test failures as HDFS-9034 did. I recommend instead to swap lines in HeartbeatManager.register() without modifying addDatanode() . Like this: synchronized void register( final DatanodeDescriptor d) { if (!d.isAlive) { - addDatanode(d); d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null ); + addDatanode(d); } }
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Konstantin Shvachko thanks for great review and ideas.. Updated patches for trunk and branch-2.7.Kindly review.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Konstantin Shvachko thanks for great review and ideas.. Updated patches for trunk and branch-2.7 .Kindly review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 7s trunk passed
        +1 compile 0m 48s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 0m 52s trunk passed
        -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 40s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 50s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 65m 57s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        90m 55s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.server.namenode.TestDeadDatanode



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878163/HDFS-11896-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux de075ed37be3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c21c260
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20350/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20350/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 7s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 52s trunk passed -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed       Patch Compile Tests +1 mvninstall 0m 50s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 65m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 90m 55s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.server.namenode.TestDeadDatanode Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878163/HDFS-11896-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux de075ed37be3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c21c260 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20350/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20350/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20350/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        The patches look really good. But some test failures are definitely related to the change. I was looking at TestDecommissioningStatus. Did not get into details, but it looks like adminState is getting wiped out for dead nodes, while dead nodes should remain decommissioned after death.
        Brahma Reddy Battula, so if you say it's harder to fix, we can go back to your previous patch for trunk, and file a followup jira for code reuse? I may have proposed a too radical change.

        Show
        shv Konstantin Shvachko added a comment - The patches look really good. But some test failures are definitely related to the change. I was looking at TestDecommissioningStatus. Did not get into details, but it looks like adminState is getting wiped out for dead nodes, while dead nodes should remain decommissioned after death. Brahma Reddy Battula , so if you say it's harder to fix, we can go back to your previous patch for trunk, and file a followup jira for code reuse? I may have proposed a too radical change.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        I think..Problem with storage reports are getting empty..Uploaded the patch.kindly check once..

        Show
        brahmareddy Brahma Reddy Battula added a comment - I think..Problem with storage reports are getting empty..Uploaded the patch.kindly check once..
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 16m 12s trunk passed
        +1 compile 0m 52s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 0m 52s trunk passed
        -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 50s the patch passed
        +1 javac 0m 50s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191)
        +1 mvnsite 0m 52s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 45s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 64m 41s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        92m 57s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878291/HDFS-11896-004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb15fea875bf 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 44350fd
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20363/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20363/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 16m 12s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 52s trunk passed -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 52s the patch passed +1 compile 0m 50s the patch passed +1 javac 0m 50s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191) +1 mvnsite 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 64m 41s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 92m 57s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878291/HDFS-11896-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb15fea875bf 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 44350fd Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20363/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20363/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20363/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 3m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 59s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 38s trunk passed
        +1 mvnsite 0m 55s trunk passed
        -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 48s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 36s the patch passed
              Other Tests
        -1 unit 73m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        102m 23s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestPread



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878291/HDFS-11896-004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 881dbeb531fc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 44350fd
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20364/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20364/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 3m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 59s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 55s trunk passed -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 48s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 0 fixed = 192 total (was 191) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 36s the patch passed       Other Tests -1 unit 73m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 102m 23s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestPread Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878291/HDFS-11896-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 881dbeb531fc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 44350fd Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20364/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20364/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20364/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        TestPread and TestDecommissioningStatus.testDecommissionStatus both can be random failure.. will raise seperate jira to track this.

        Show
        brahmareddy Brahma Reddy Battula added a comment - TestPread and TestDecommissioningStatus.testDecommissionStatus both can be random failure.. will raise seperate jira to track this.
        Hide
        shv Konstantin Shvachko added a comment -

        That's it, good catch. I have only one comment for the patch. In testNonDFSUsedONDeadNodeReReg() you can calculate dnDesc = cluster.getNamesystem(0).getBlockManager().getDatanodeManager().getDatanode(dn.getDatanodeId());. That way you will not need to introduce DatanodeManager.getDatanodes() in branch-2.7.
        +1 otherwise.

        Branch 2.7 patch comments:

        1. Patch should also be updated to not empty storages
        2. Few empty line problems:
          • Add empty lines before and after isHeartbeatedSinceRegistration()
          • Remove empty line after testNonDFSUsedONDeadNodeReReg()
          • Remove empty line inside register(
          • Remove new empty line before setHeartbeatsDisabledForTests()
        3. Remove getDatanodes(). Don't need it if you calculate dnDesc as above.

        It would be good to run Jenkins on 2.7 patch. So attach it last, please. Thanks for following up on this.

        Show
        shv Konstantin Shvachko added a comment - That's it, good catch. I have only one comment for the patch. In testNonDFSUsedONDeadNodeReReg() you can calculate dnDesc = cluster.getNamesystem(0).getBlockManager().getDatanodeManager().getDatanode(dn.getDatanodeId()); . That way you will not need to introduce DatanodeManager.getDatanodes() in branch-2.7. +1 otherwise. Branch 2.7 patch comments: Patch should also be updated to not empty storages Few empty line problems: Add empty lines before and after isHeartbeatedSinceRegistration() Remove empty line after testNonDFSUsedONDeadNodeReReg() Remove empty line inside register( Remove new empty line before setHeartbeatsDisabledForTests() Remove getDatanodes() . Don't need it if you calculate dnDesc as above. It would be good to run Jenkins on 2.7 patch. So attach it last, please. Thanks for following up on this.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Updated the trunk patch.

        It would be good to run Jenkins on 2.7 patch. So attach it last, please. Thanks for following up on this.

        branch-2.7 pre-commit jenkins build was failing even I sent mail in mailing list (here)..Just I am waiting for reply hence I didn't uploaded branch-2.7 patch before.. As of now nobody reply..

        Since there is plan for branch-2.7,this need to fixed ASAP.
        Bytheway I updated the wiki for 2.7.4 release status

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Updated the trunk patch. It would be good to run Jenkins on 2.7 patch. So attach it last, please. Thanks for following up on this. branch-2.7 pre-commit jenkins build was failing even I sent mail in mailing list ( here )..Just I am waiting for reply hence I didn't uploaded branch-2.7 patch before.. As of now nobody reply.. Since there is plan for branch-2.7 ,this need to fixed ASAP. Bytheway I updated the wiki for 2.7.4 release status
        Hide
        shv Konstantin Shvachko added a comment -

        Added dev@yetus to the thread.
        Could you please attach the 2.7 patch anyways.

        Show
        shv Konstantin Shvachko added a comment - Added dev@yetus to the thread. Could you please attach the 2.7 patch anyways.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Added dev@yetus to the thread.

        thanks a lot.

        Could you please attach the 2.7 patch anyways.

        Uploaded.Hope I addressed all your comments.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Added dev@yetus to the thread. thanks a lot. Could you please attach the 2.7 patch anyways. Uploaded.Hope I addressed all your comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        -1 mvninstall 15m 24s root in trunk failed.
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 0m 53s trunk passed
        -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191)
        +1 mvnsite 0m 50s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 44s the patch passed
        +1 javadoc 0m 37s the patch passed
              Other Tests
        -1 unit 63m 54s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        91m 4s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.namenode.TestDeadDatanode



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878433/HDFS-11896-005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4d05706af9b4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 465c213
        Default Java 1.8.0_131
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/branch-mvninstall-root.txt
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20384/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20384/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests -1 mvninstall 15m 24s root in trunk failed. +1 compile 0m 47s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 53s trunk passed -1 findbugs 1m 38s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 191 unchanged - 0 fixed = 193 total (was 191) +1 mvnsite 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 44s the patch passed +1 javadoc 0m 37s the patch passed       Other Tests -1 unit 63m 54s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 91m 4s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.namenode.TestDeadDatanode Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878433/HDFS-11896-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4d05706af9b4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 465c213 Default Java 1.8.0_131 mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/branch-mvninstall-root.txt findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20384/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20384/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20384/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shv Konstantin Shvachko added a comment -

        Let's make sure TestDeadDatanode.testNonDFSUsedONDeadNodeReReg() is not failing.

        Show
        shv Konstantin Shvachko added a comment - Let's make sure TestDeadDatanode.testNonDFSUsedONDeadNodeReReg() is not failing.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Hope this should work.ORelse i should use simulate capacities.. Sorry, as i can't expect jenkins data, it was failing earlier.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Hope this should work.ORelse i should use simulate capacities.. Sorry, as i can't expect jenkins data, it was failing earlier.
        Hide
        shv Konstantin Shvachko added a comment -

        This is still failing locally for me:

        java.lang.AssertionError: NonDFS should include actual DN NonDFSUsed expected:<245913960448> but was:<245914312704>
        	at org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode.testNonDFSUsedONDeadNodeReReg(TestDeadDatanode.java:222)
        

        Seems that nonDfsUsed cannot be exactly the same at different times, because somebody is always writing to disk, including this test logging.

        Show
        shv Konstantin Shvachko added a comment - This is still failing locally for me: java.lang.AssertionError: NonDFS should include actual DN NonDFSUsed expected:<245913960448> but was:<245914312704> at org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode.testNonDFSUsedONDeadNodeReReg(TestDeadDatanode.java:222) Seems that nonDfsUsed cannot be exactly the same at different times, because somebody is always writing to disk, including this test logging.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Used the simulated capacities, hope this should work...It will always pass in my local might be I will be not running tests parallel Or jenkins might create some data
        Konstantin Shvachko Sorry,again...I knew, you efforts for branch-2.7 release even I am looking for this.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Used the simulated capacities, hope this should work...It will always pass in my local might be I will be not running tests parallel Or jenkins might create some data Konstantin Shvachko Sorry,again...I knew, you efforts for branch-2.7 release even I am looking for this.
        Hide
        shv Konstantin Shvachko added a comment -

        +1 on the 007 patch.

        Show
        shv Konstantin Shvachko added a comment - +1 on the 007 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 4m 46s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 19m 28s trunk passed
        +1 compile 1m 7s trunk passed
        +1 checkstyle 0m 51s trunk passed
        +1 mvnsite 1m 26s trunk passed
        -1 findbugs 2m 25s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 56s trunk passed
              Patch Compile Tests
        +1 mvninstall 1m 20s the patch passed
        +1 compile 1m 10s the patch passed
        +1 javac 1m 10s the patch passed
        -0 checkstyle 0m 46s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 191 unchanged - 0 fixed = 194 total (was 191)
        +1 mvnsite 1m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 30s the patch passed
        +1 javadoc 0m 49s the patch passed
              Other Tests
        -1 unit 97m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        138m 30s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878727/HDFS-11896-007.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1d38c8ab45fc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c98201b
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20403/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20403/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 4m 46s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 28s trunk passed +1 compile 1m 7s trunk passed +1 checkstyle 0m 51s trunk passed +1 mvnsite 1m 26s trunk passed -1 findbugs 2m 25s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 56s trunk passed       Patch Compile Tests +1 mvninstall 1m 20s the patch passed +1 compile 1m 10s the patch passed +1 javac 1m 10s the patch passed -0 checkstyle 0m 46s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 191 unchanged - 0 fixed = 194 total (was 191) +1 mvnsite 1m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 30s the patch passed +1 javadoc 0m 49s the patch passed       Other Tests -1 unit 97m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 138m 30s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878727/HDFS-11896-007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1d38c8ab45fc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c98201b Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20403/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20403/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20403/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the branch-2.7 patch. Kindly review.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the branch-2.7 patch. Kindly review.
        Hide
        shv Konstantin Shvachko added a comment -

        For branch-2.7-005 patch there is only one problem that it does not fail without the changes, which means it is not actually testing the condition we fix. Is it because of simulated capacities or did I miss something?

        Show
        shv Konstantin Shvachko added a comment - For branch-2.7-005 patch there is only one problem that it does not fail without the changes, which means it is not actually testing the condition we fix. Is it because of simulated capacities or did I miss something?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Oh,Yes.It's because of simulated only..will check alternative for this.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Oh,Yes.It's because of simulated only..will check alternative for this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 3m 21s Docker failed to build yetus/hadoop:67e87c9.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20421/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 3m 21s Docker failed to build yetus/hadoop:67e87c9. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20421/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 0s Docker failed to build yetus/hadoop:67e87c9.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20422/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 0s Docker failed to build yetus/hadoop:67e87c9. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20422/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 0s Docker failed to build yetus/hadoop:67e87c9.



        Subsystem Report/Notes
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20423/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 0s Docker failed to build yetus/hadoop:67e87c9. Subsystem Report/Notes JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878793/HDFS-11896-branch-2.7-005.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20423/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the work Brahma Reddy Battula.

        I modified the code base to use non-simulated capacity, and added an intermediate variable for the nonDFS used capacity after one DN is dead but before it registers.

          @Test
          public void testNonDFSUsedONDeadNodeReReg() throws Exception {
            Configuration conf = new HdfsConfiguration();
            conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);
            conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 1);
            conf.setInt(DFSConfigKeys.DFS_NAMENODE_STALE_DATANODE_INTERVAL_KEY,
                6 * 1000);
            long capacity = 5000L;
            long[] capacities = new long[]{ 4 * capacity, 4 * capacity };
            try {
              cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build();
              long initialCapacity = cluster.getNamesystem(0).getCapacityTotal();
              long nonDFS = cluster.getNamesystem(0).getNonDfsUsedSpace();
              assertTrue(initialCapacity > 0);
              DataNode dn1 = cluster.getDataNodes().get(0);
              DataNode dn2 = cluster.getDataNodes().get(1);
              final DatanodeDescriptor dn2Desc = cluster.getNamesystem(0)
                  .getBlockManager().getDatanodeManager()
                  .getDatanode(dn2.getDatanodeId());
              dn1.setHeartbeatsDisabledForTests(true);
              cluster.setDataNodeDead(dn1.getDatanodeId());
              assertEquals("Capacity shouldn't include DeadNode", dn2Desc.getCapacity(),
                  cluster.getNamesystem(0).getCapacityTotal());
              long nonDFSWithDeadDN = cluster.getNamesystem(0).getNonDfsUsedSpace();
              assertEquals("NonDFS-used shouldn't include DeadNode",
                  dn2Desc.getNonDfsUsed(), nonDFSWithDeadDN);
              // Wait for re-registration and heartbeat
              dn1.setHeartbeatsDisabledForTests(false);
              final DatanodeDescriptor dn1Desc = cluster.getNamesystem(0)
                  .getBlockManager().getDatanodeManager()
                  .getDatanode(dn1.getDatanodeId());
              GenericTestUtils.waitFor(new Supplier<Boolean>() {
        
                @Override
                public Boolean get() {
                  return dn1Desc.isAlive && dn1Desc.isHeartbeatedSinceRegistration();
                }
              }, 100, 5000);
              assertEquals("Capacity should be 0 after all DNs dead", initialCapacity,
                  cluster.getNamesystem(0).getCapacityTotal());
              long nonDfsAfterReg = dn1Desc.getNonDfsUsed() + dn2Desc.getNonDfsUsed();
              LOG.info("nonDFS=" + nonDFS + ",nonDFSWithDeadDN=" + nonDFSWithDeadDN +
                      ",nonDfsAfterReg=" + nonDfsAfterReg);
              assertEquals("NonDFS should include actual DN NonDFSUsed", nonDFS,
                  nonDfsAfterReg);
            } finally {
              if (cluster != null) {
                cluster.shutdown();
              }
            }
          }
        

        Actually I don't see a clear difference between the behavior with and without the patch. Did you observe that the non-dfsUsed number actually doubled? And "doubled" here means 2x the amount of non-dfsUsed on the dead DN was added to the Namesystem overall statics? If so do you mind updating the JIRA description to be more accurate? Thanks.

        // Without patch
        nonDFS=884109852672,nonDFSWithDeadDN=442054926336,nonDfsAfterReg=884110409728
        nonDFS=884111327232,nonDFSWithDeadDN=442055663616,nonDfsAfterReg=884112097280
        nonDFS=884115406848,nonDFSWithDeadDN=442057703424,nonDfsAfterReg=884116340736
        
        // With patch
        nonDFS=884110589952,nonDFSWithDeadDN=442055311360,nonDfsAfterReg=884111163392
        nonDFS=884116471808,nonDFSWithDeadDN=442058235904,nonDfsAfterReg=884115488768
        nonDFS=884118700032,nonDFSWithDeadDN=442059350016,nonDfsAfterReg=884119486464
        

        Minor: long[] capacities is unused.

        Show
        zhz Zhe Zhang added a comment - Thanks for the work Brahma Reddy Battula . I modified the code base to use non-simulated capacity, and added an intermediate variable for the nonDFS used capacity after one DN is dead but before it registers. @Test public void testNonDFSUsedONDeadNodeReReg() throws Exception { Configuration conf = new HdfsConfiguration(); conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1); conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 1); conf.setInt(DFSConfigKeys.DFS_NAMENODE_STALE_DATANODE_INTERVAL_KEY, 6 * 1000); long capacity = 5000L; long [] capacities = new long []{ 4 * capacity, 4 * capacity }; try { cluster = new MiniDFSCluster.Builder(conf).numDataNodes(2).build(); long initialCapacity = cluster.getNamesystem(0).getCapacityTotal(); long nonDFS = cluster.getNamesystem(0).getNonDfsUsedSpace(); assertTrue(initialCapacity > 0); DataNode dn1 = cluster.getDataNodes().get(0); DataNode dn2 = cluster.getDataNodes().get(1); final DatanodeDescriptor dn2Desc = cluster.getNamesystem(0) .getBlockManager().getDatanodeManager() .getDatanode(dn2.getDatanodeId()); dn1.setHeartbeatsDisabledForTests( true ); cluster.setDataNodeDead(dn1.getDatanodeId()); assertEquals( "Capacity shouldn't include DeadNode" , dn2Desc.getCapacity(), cluster.getNamesystem(0).getCapacityTotal()); long nonDFSWithDeadDN = cluster.getNamesystem(0).getNonDfsUsedSpace(); assertEquals( "NonDFS-used shouldn't include DeadNode" , dn2Desc.getNonDfsUsed(), nonDFSWithDeadDN); // Wait for re-registration and heartbeat dn1.setHeartbeatsDisabledForTests( false ); final DatanodeDescriptor dn1Desc = cluster.getNamesystem(0) .getBlockManager().getDatanodeManager() .getDatanode(dn1.getDatanodeId()); GenericTestUtils.waitFor( new Supplier< Boolean >() { @Override public Boolean get() { return dn1Desc.isAlive && dn1Desc.isHeartbeatedSinceRegistration(); } }, 100, 5000); assertEquals( "Capacity should be 0 after all DNs dead" , initialCapacity, cluster.getNamesystem(0).getCapacityTotal()); long nonDfsAfterReg = dn1Desc.getNonDfsUsed() + dn2Desc.getNonDfsUsed(); LOG.info( "nonDFS=" + nonDFS + ",nonDFSWithDeadDN=" + nonDFSWithDeadDN + ",nonDfsAfterReg=" + nonDfsAfterReg); assertEquals( "NonDFS should include actual DN NonDFSUsed" , nonDFS, nonDfsAfterReg); } finally { if (cluster != null ) { cluster.shutdown(); } } } Actually I don't see a clear difference between the behavior with and without the patch. Did you observe that the non-dfsUsed number actually doubled? And "doubled" here means 2x the amount of non-dfsUsed on the dead DN was added to the Namesystem overall statics? If so do you mind updating the JIRA description to be more accurate? Thanks. // Without patch nonDFS=884109852672,nonDFSWithDeadDN=442054926336,nonDfsAfterReg=884110409728 nonDFS=884111327232,nonDFSWithDeadDN=442055663616,nonDfsAfterReg=884112097280 nonDFS=884115406848,nonDFSWithDeadDN=442057703424,nonDfsAfterReg=884116340736 // With patch nonDFS=884110589952,nonDFSWithDeadDN=442055311360,nonDfsAfterReg=884111163392 nonDFS=884116471808,nonDFSWithDeadDN=442058235904,nonDfsAfterReg=884115488768 nonDFS=884118700032,nonDFSWithDeadDN=442059350016,nonDfsAfterReg=884119486464 Minor: long[] capacities is unused.
        Hide
        zhz Zhe Zhang added a comment -

        Uploading a new branch-2.7 patch to fix the test. Now it passes with the patch and fails without.

        Now the test test Namesystem's NonDfsUsed after dead DN registers, and compares it with the sum from the two DN descriptor's NonDfsUsed.

        Show
        zhz Zhe Zhang added a comment - Uploading a new branch-2.7 patch to fix the test. Now it passes with the patch and fails without. Now the test test Namesystem's NonDfsUsed after dead DN registers, and compares it with the sum from the two DN descriptor's NonDfsUsed.
        Hide
        shv Konstantin Shvachko added a comment - - edited

        Actually let me just commit Zhe Zhang's patch, tahnks. +1
        Will update the trunk patch accordingly and commit.

        Show
        shv Konstantin Shvachko added a comment - - edited Actually let me just commit Zhe Zhang 's patch, tahnks. +1 Will update the trunk patch accordingly and commit.
        Hide
        shv Konstantin Shvachko added a comment -

        Updated both patches. Removed now unused variable nonDFS

        Show
        shv Konstantin Shvachko added a comment - Updated both patches. Removed now unused variable nonDFS
        Hide
        shv Konstantin Shvachko added a comment -

        I just committed this to trunk, and branches 2, 2.8, and 2.7.
        Thank you Brahma Reddy Battula and Zhe Zhang.

        Show
        shv Konstantin Shvachko added a comment - I just committed this to trunk, and branches 2, 2.8, and 2.7. Thank you Brahma Reddy Battula and Zhe Zhang .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 2s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.7 Compile Tests
        +1 mvninstall 8m 55s branch-2.7 passed
        +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_131
        +1 compile 1m 7s branch-2.7 passed with JDK v1.7.0_131
        +1 checkstyle 0m 28s branch-2.7 passed
        +1 mvnsite 1m 4s branch-2.7 passed
        +1 findbugs 3m 14s branch-2.7 passed
        +1 javadoc 1m 5s branch-2.7 passed with JDK v1.8.0_131
        +1 javadoc 1m 55s branch-2.7 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 54s the patch passed
        +1 compile 1m 10s the patch passed with JDK v1.8.0_131
        +1 javac 1m 10s the patch passed
        +1 compile 1m 2s the patch passed with JDK v1.7.0_131
        +1 javac 1m 2s the patch passed
        -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 264 unchanged - 0 fixed = 265 total (was 264)
        +1 mvnsite 0m 58s the patch passed
        -1 whitespace 0m 0s The patch has 64 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 3m 15s the patch passed
        +1 javadoc 0m 59s the patch passed with JDK v1.8.0_131
        +1 javadoc 1m 46s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 53m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        148m 42s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.hdfs.TestDecommission
          hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
        JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestDatanodeRegistration
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.web.TestWebHdfsTokens
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:67e87c9
        JIRA Issue HDFS-11896
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879203/HDFS-11896-branch-2.7-006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 784d9e8bba5f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / b516235
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20436/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20436/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 2s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.7 Compile Tests +1 mvninstall 8m 55s branch-2.7 passed +1 compile 1m 11s branch-2.7 passed with JDK v1.8.0_131 +1 compile 1m 7s branch-2.7 passed with JDK v1.7.0_131 +1 checkstyle 0m 28s branch-2.7 passed +1 mvnsite 1m 4s branch-2.7 passed +1 findbugs 3m 14s branch-2.7 passed +1 javadoc 1m 5s branch-2.7 passed with JDK v1.8.0_131 +1 javadoc 1m 55s branch-2.7 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 54s the patch passed +1 compile 1m 10s the patch passed with JDK v1.8.0_131 +1 javac 1m 10s the patch passed +1 compile 1m 2s the patch passed with JDK v1.7.0_131 +1 javac 1m 2s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 264 unchanged - 0 fixed = 265 total (was 264) +1 mvnsite 0m 58s the patch passed -1 whitespace 0m 0s The patch has 64 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 15s the patch passed +1 javadoc 0m 59s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 46s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 53m 55s hadoop-hdfs in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 148m 42s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.hdfs.TestDecommission   hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots JDK v1.7.0_131 Failed junit tests hadoop.hdfs.TestDatanodeRegistration   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.web.TestWebHdfsTokens   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:67e87c9 JIRA Issue HDFS-11896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879203/HDFS-11896-branch-2.7-006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 784d9e8bba5f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / b516235 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20436/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20436/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20436/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12059 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12059/)
        HDFS-11896. Non-dfsUsed will be doubled on dead node re-registration. (shv: rev c4a85c694fae3f814ab4e7f3c172da1df0e0e353)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12059 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12059/ ) HDFS-11896 . Non-dfsUsed will be doubled on dead node re-registration. (shv: rev c4a85c694fae3f814ab4e7f3c172da1df0e0e353) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Konstantin Shvachko thanks a lot for continous review and commit.. Zhe Zhang thanks for updating patch,I was busy with other things.

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Konstantin Shvachko thanks a lot for continous review and commit.. Zhe Zhang thanks for updating patch,I was busy with other things.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        IMO, this should be backported to branch-2.8.2 as well..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - IMO, this should be backported to branch-2.8.2 as well..?
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Pushed to branch-2.8.2 as well. Compiled the ran the testcase locally.

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        
        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.49 sec - in org.apache.hadoop.hdfs.server.namenode.Te
        stDeadDatanode
        Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0
        
        Results :
        
        Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
        
        Show
        brahmareddy Brahma Reddy Battula added a comment - Pushed to branch-2.8.2 as well. Compiled the ran the testcase locally. ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.49 sec - in org.apache.hadoop.hdfs.server.namenode.Te stDeadDatanode Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Results : Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
        Hide
        djp Junping Du added a comment -

        Thanks Brahma Reddy Battula for identify this not landing on branch-2.8.2. Drop 2.8.3 in fix version.

        Show
        djp Junping Du added a comment - Thanks Brahma Reddy Battula for identify this not landing on branch-2.8.2. Drop 2.8.3 in fix version.

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            brahmareddy Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development