Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11146

Excess replicas will not be deleted until all storages's FBR received after failover

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Excess replicas will not be deleted until all storages's FBR received after failover.

      Thinking following soultion can help.

      Solution:

      I think after failover, As DNs aware of failover ,so they can send another block report (FBR) irrespective of interval.May be some shuffle can be done, similar to initial delay.

      1. HDFS-11146.patch
        19 kB
        Brahma Reddy Battula
      2. HDFS-11146-002.patch
        20 kB
        Brahma Reddy Battula
      3. HDFS-11146-003.patch
        22 kB
        Brahma Reddy Battula

        Activity

        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Any thoughts on this..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Any thoughts on this..?
        Hide
        kihwal Kihwal Lee added a comment -

        NN could tell datanodes to send FBR in a heartbeat response. This way, NN can decide when to get a FBR from whom, instead of DNs randomly sending.

        Show
        kihwal Kihwal Lee added a comment - NN could tell datanodes to send FBR in a heartbeat response. This way, NN can decide when to get a FBR from whom, instead of DNs randomly sending.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Kihwal Lee thanks for taking look.

        NN could tell datanodes to send FBR in a heartbeat response.

        Yes,it's good idea. we can do this but we should not ask all at once.this needs to take care.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Kihwal Lee thanks for taking look. NN could tell datanodes to send FBR in a heartbeat response. Yes,it's good idea. we can do this but we should not ask all at once.this needs to take care.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploading the draft patch..
        Need to limit number of block reports requested at a time.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploading the draft patch.. Need to limit number of block reports requested at a time.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 37s trunk passed
        +1 compile 0m 57s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 55s trunk passed
        +1 javadoc 0m 42s trunk passed
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 54s the patch passed
        +1 cc 0m 54s the patch passed
        +1 javac 0m 54s the patch passed
        -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 17 new + 501 unchanged - 0 fixed = 518 total (was 501)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 56s the patch passed
        +1 javadoc 0m 38s the patch passed
        -1 unit 106m 16s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        127m 29s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.fs.TestSymlinkHdfsFileSystem
        Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11146
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840419/HDFS-11146.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux aaeba891da31 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / eb0a483
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17659/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17659/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17659/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17659/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 55s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 54s the patch passed +1 cc 0m 54s the patch passed +1 javac 0m 54s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 17 new + 501 unchanged - 0 fixed = 518 total (was 501) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 56s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 106m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 127m 29s Reason Tests Failed junit tests hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.fs.TestSymlinkHdfsFileSystem Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11146 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840419/HDFS-11146.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux aaeba891da31 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / eb0a483 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17659/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17659/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17659/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17659/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Test failures are unrelated,Kihwal Lee can you take look at the patch,if you get chance.?

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Test failures are unrelated, Kihwal Lee can you take look at the patch,if you get chance.?
        Hide
        kihwal Kihwal Lee added a comment -

        Sorry, I am busy and won't be able to review it properly soon. I will probably get to it next week. I would pay close attention to compatibility, interactions with block report lease, etc.

        Show
        kihwal Kihwal Lee added a comment - Sorry, I am busy and won't be able to review it properly soon. I will probably get to it next week. I would pay close attention to compatibility, interactions with block report lease, etc.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Nope.. thanks a lot for your close attention on this issue..

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Nope.. thanks a lot for your close attention on this issue..
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Kihwal Lee if you get chance, can you please review.? As of now,we can disable this by default..?

        Show
        brahmareddy Brahma Reddy Battula added a comment - Kihwal Lee if you get chance, can you please review.? As of now,we can disable this by default..?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 8s HDFS-11146 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-11146
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840419/HDFS-11146.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19812/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 8s HDFS-11146 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-11146 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840419/HDFS-11146.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19812/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Brahma Reddy Battula: seems like this patch doesn't apply anymore.
        Can you please update the patch. In the meantime, I will try to review.
        Thanks !

        Show
        shahrs87 Rushabh S Shah added a comment - Brahma Reddy Battula : seems like this patch doesn't apply anymore. Can you please update the patch. In the meantime, I will try to review. Thanks !
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Uploaded the latest patch.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Uploaded the latest patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 59s trunk passed
        +1 compile 0m 49s trunk passed
        +1 checkstyle 0m 48s trunk passed
        +1 mvnsite 0m 52s trunk passed
        -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 40s trunk passed
              Patch Compile Tests
        -1 mvninstall 0m 25s hadoop-hdfs in the patch failed.
        -1 compile 0m 25s hadoop-hdfs in the patch failed.
        -1 cc 0m 25s hadoop-hdfs in the patch failed.
        -1 javac 0m 25s hadoop-hdfs in the patch failed.
        -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 15 new + 882 unchanged - 0 fixed = 897 total (was 882)
        -1 mvnsite 0m 27s hadoop-hdfs in the patch failed.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 25s hadoop-hdfs in the patch failed.
        -1 javadoc 0m 40s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9)
              Other Tests
        -1 unit 0m 28s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        24m 17s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11146
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877086/HDFS-11146-002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 753d408f0814 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b61ab85
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
        compile https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        cc https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
        javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20258/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20258/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 59s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 0m 52s trunk passed -1 findbugs 1m 42s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed       Patch Compile Tests -1 mvninstall 0m 25s hadoop-hdfs in the patch failed. -1 compile 0m 25s hadoop-hdfs in the patch failed. -1 cc 0m 25s hadoop-hdfs in the patch failed. -1 javac 0m 25s hadoop-hdfs in the patch failed. -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 15 new + 882 unchanged - 0 fixed = 897 total (was 882) -1 mvnsite 0m 27s hadoop-hdfs in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 25s hadoop-hdfs in the patch failed. -1 javadoc 0m 40s hadoop-hdfs-project_hadoop-hdfs generated 5 new + 9 unchanged - 0 fixed = 14 total (was 9)       Other Tests -1 unit 0m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 24m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11146 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877086/HDFS-11146-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 753d408f0814 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b61ab85 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html mvninstall https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt compile https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt cc https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt javac https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt mvnsite https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt javadoc https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20258/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20258/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20258/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Looks newly added class is missed in v2.. updated now.

        Show
        brahmareddy Brahma Reddy Battula added a comment - Looks newly added class is missed in v2.. updated now.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 15s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 45s trunk passed
        +1 mvnsite 0m 56s trunk passed
        -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 42s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 51s the patch passed
        +1 cc 0m 51s the patch passed
        -1 javac 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 411 unchanged - 0 fixed = 412 total (was 411)
        -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 882 unchanged - 0 fixed = 890 total (was 882)
        +1 mvnsite 0m 54s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 40s the patch passed
              Other Tests
        -1 unit 66m 47s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        92m 46s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.tools.TestHdfsConfigFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11146
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877211/HDFS-11146-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux ce8b52ba1ad5 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 43f0503
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20268/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20268/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 15s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 0m 56s trunk passed -1 findbugs 1m 47s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 42s trunk passed       Patch Compile Tests +1 mvninstall 0m 51s the patch passed +1 compile 0m 51s the patch passed +1 cc 0m 51s the patch passed -1 javac 0m 51s hadoop-hdfs-project_hadoop-hdfs generated 1 new + 411 unchanged - 0 fixed = 412 total (was 411) -0 checkstyle 0m 44s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 882 unchanged - 0 fixed = 890 total (was 882) +1 mvnsite 0m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 40s the patch passed       Other Tests -1 unit 66m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 92m 46s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11146 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877211/HDFS-11146-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ce8b52ba1ad5 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 43f0503 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html javac https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20268/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20268/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20268/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        brahmareddy Brahma Reddy Battula added a comment - - edited

        Rushabh S Shah patch is ready for review..Will update checksystyle and testcase failure once after review is done.Kindly review.

        Show
        brahmareddy Brahma Reddy Battula added a comment - - edited Rushabh S Shah patch is ready for review..Will update checksystyle and testcase failure once after review is done.Kindly review.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        Rushabh S Shah patch is ready for review..

        Thanks ! Will do it today or max tomorrow.

        Show
        shahrs87 Rushabh S Shah added a comment - Rushabh S Shah patch is ready for review.. Thanks ! Will do it today or max tomorrow.
        Hide
        shahrs87 Rushabh S Shah added a comment -

        I started reviewing the patch.
        I have one high level question.

        Yes,it's good idea. we can do this but we should not ask all at once.this needs to take care.

        Even in the latest patch, this will happen, correct ?
        After a failover, namenode will ask for block report from all the nodes at once.
        This will create a block report storm on namenode.
        Correct me if I am wrong.
        Sorry for reviewing so late.

        Show
        shahrs87 Rushabh S Shah added a comment - I started reviewing the patch. I have one high level question. Yes,it's good idea. we can do this but we should not ask all at once.this needs to take care. Even in the latest patch, this will happen, correct ? After a failover, namenode will ask for block report from all the nodes at once. This will create a block report storm on namenode. Correct me if I am wrong. Sorry for reviewing so late.
        Hide
        daryn Daryn Sharp added a comment - - edited

        Yes, this appears it would destroy the NN with FBRs. I'd rather see the existing DNA_REGISTER command, rather than a new command, be used to indirectly solicit a FBR. The register will schedule the FBR request a short time in the future and utilize the existing FBR leases to avoid the storm.

        I'd rather not have the common case for heartbeat processing taking the extra expense for a rare use case of failover. It would be better for the heartbeat monitor to introduce the expense on a less frequent basis. It can call setForceRegistration on the datanode descriptor and the next heartbeat will trigger the DNA_REGISTER.

        Show
        daryn Daryn Sharp added a comment - - edited Yes, this appears it would destroy the NN with FBRs. I'd rather see the existing DNA_REGISTER command, rather than a new command, be used to indirectly solicit a FBR. The register will schedule the FBR request a short time in the future and utilize the existing FBR leases to avoid the storm. I'd rather not have the common case for heartbeat processing taking the extra expense for a rare use case of failover. It would be better for the heartbeat monitor to introduce the expense on a less frequent basis. It can call setForceRegistration on the datanode descriptor and the next heartbeat will trigger the DNA_REGISTER.

          People

          • Assignee:
            brahmareddy Brahma Reddy Battula
            Reporter:
            brahmareddy Brahma Reddy Battula
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development