Details

      Description

      This jira is to visit the replication based config keys and deprecate them(if necessary) in order to make it more meaningful.

      Please refer discussion thread

      1. HDFS-10368-00.patch
        63 kB
        Rakesh R
      2. HDFS-10368-01.patch
        65 kB
        Rakesh R
      3. HDFS-10368-02.patch
        77 kB
        Rakesh R
      4. HDFS-10368-03.patch
        79 kB
        Rakesh R
      5. HDFS-10368-04.patch
        79 kB
        Rakesh R

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Rakesh, do you still plan to work on this JIRA? I see it's labeled "hdfs-ec-3.0-must-do".

        Show
        andrew.wang Andrew Wang added a comment - Hi Rakesh, do you still plan to work on this JIRA? I see it's labeled "hdfs-ec-3.0-must-do".
        Hide
        rakeshr Rakesh R added a comment - - edited

        Thanks Andrew Wang for the reminder. Earlier we had identified one property and deprecated it as part of HDFS-9869.

        https://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
        dfs.replication.pending.timeout.sec	         =>   dfs.namenode.reconstruction.pending.timeout-sec
        dfs.namenode.replication.pending.timeout-sec     =>   dfs.namenode.reconstruction.pending.timeout-sec
        

        Like mentioned, this jira to revisit the replication related configs, following are few replication config items. I remember somewhere we discussed to consider EC internal blocks are a special form of replicas, imho documentation would be sufficient rather than renaming these configs to reconstruction or redundancy. Do we have docs conveying the same? Makes sense to you?

        dfs.namenode.replication.considerLoad
        dfs.namenode.replication.considerLoad.factor
        dfs.namenode.replication.min
        dfs.namenode.replication.interval
        dfs.namenode.replication.work.multiplier.per.iteration
        dfs.namenode.replication.max-streams
        dfs.namenode.replication.max-streams-hard-limit
        
        Show
        rakeshr Rakesh R added a comment - - edited Thanks Andrew Wang for the reminder. Earlier we had identified one property and deprecated it as part of HDFS-9869 . https: //hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html dfs.replication.pending.timeout.sec => dfs.namenode.reconstruction.pending.timeout-sec dfs.namenode.replication.pending.timeout-sec => dfs.namenode.reconstruction.pending.timeout-sec Like mentioned, this jira to revisit the replication related configs, following are few replication config items. I remember somewhere we discussed to consider EC internal blocks are a special form of replicas , imho documentation would be sufficient rather than renaming these configs to reconstruction or redundancy. Do we have docs conveying the same? Makes sense to you? dfs.namenode.replication.considerLoad dfs.namenode.replication.considerLoad.factor dfs.namenode.replication.min dfs.namenode.replication.interval dfs.namenode.replication.work.multiplier.per.iteration dfs.namenode.replication.max-streams dfs.namenode.replication.max-streams-hard-limit
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for the investigation Rakesh. I think there's little harm in adding deprecations, since the old keys will keep on working.

        Some of these seem like a simple rename+deprecation which we can handle here:

        • dfs.namenode.replication.considerLoad
        • dfs.namenode.replication.considerLoad.factor
        • dfs.namenode.replication.interval

        Some of these we should leave:

        • dfs.namenode.replication.min - I assume this only applies to replicated blocks, not striped?
        • The various safemode / maintenance mode replication thresholds, I assume there is different handling for striped blocks?

        Some seem like they need to be reworked since they no longer do what is advertised, since EC reconstruction involves reading multiple blocks:

        • dfs.namenode.replication.work.multiplier.per.iteration
        • dfs.namenode.replication.max-streams
        • dfs.namenode.replication.max-streams-hard-limit

        These last three will hopefully be addressed by HDFS-11023.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for the investigation Rakesh. I think there's little harm in adding deprecations, since the old keys will keep on working. Some of these seem like a simple rename+deprecation which we can handle here: dfs.namenode.replication.considerLoad dfs.namenode.replication.considerLoad.factor dfs.namenode.replication.interval Some of these we should leave: dfs.namenode.replication.min - I assume this only applies to replicated blocks, not striped? The various safemode / maintenance mode replication thresholds, I assume there is different handling for striped blocks? Some seem like they need to be reworked since they no longer do what is advertised, since EC reconstruction involves reading multiple blocks: dfs.namenode.replication.work.multiplier.per.iteration dfs.namenode.replication.max-streams dfs.namenode.replication.max-streams-hard-limit These last three will hopefully be addressed by HDFS-11023 .
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Andrew Wang, I will rename the following set of configurations in this jira,

        dfs.namenode.replication.considerLoad
        dfs.namenode.replication.considerLoad.factor
        dfs.namenode.replication.interval
        

        to

        dfs.namenode.redundancy.considerLoad
        dfs.namenode.redundancy.considerLoad.factor
        dfs.namenode.redundancy.interval
        
        Show
        rakeshr Rakesh R added a comment - Thanks Andrew Wang , I will rename the following set of configurations in this jira, dfs.namenode.replication.considerLoad dfs.namenode.replication.considerLoad.factor dfs.namenode.replication.interval to dfs.namenode.redundancy.considerLoad dfs.namenode.redundancy.considerLoad.factor dfs.namenode.redundancy.interval
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 40 new or modified test files.
        0 mvndep 0m 57s Maven dependency ordering for branch
        +1 mvninstall 8m 1s trunk passed
        +1 compile 7m 50s trunk passed
        +1 checkstyle 1m 54s trunk passed
        +1 mvnsite 2m 38s trunk passed
        +1 mvneclipse 0m 41s trunk passed
        +1 findbugs 4m 49s trunk passed
        +1 javadoc 1m 47s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 2m 5s the patch passed
        +1 compile 8m 2s the patch passed
        +1 javac 8m 2s the patch passed
        -0 checkstyle 1m 53s root: The patch generated 7 new + 1623 unchanged - 9 fixed = 1630 total (was 1632)
        +1 mvnsite 2m 33s the patch passed
        +1 mvneclipse 0m 41s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 5m 26s the patch passed
        +1 javadoc 1m 50s the patch passed
        +1 unit 8m 42s hadoop-common in the patch passed.
        +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
        -1 unit 57m 29s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        120m 30s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestCrcCorruption



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10368
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837172/HDFS-10368-00.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux c62ed277614b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0aafc12
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17433/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17433/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 40 new or modified test files. 0 mvndep 0m 57s Maven dependency ordering for branch +1 mvninstall 8m 1s trunk passed +1 compile 7m 50s trunk passed +1 checkstyle 1m 54s trunk passed +1 mvnsite 2m 38s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 4m 49s trunk passed +1 javadoc 1m 47s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 2m 5s the patch passed +1 compile 8m 2s the patch passed +1 javac 8m 2s the patch passed -0 checkstyle 1m 53s root: The patch generated 7 new + 1623 unchanged - 9 fixed = 1630 total (was 1632) +1 mvnsite 2m 33s the patch passed +1 mvneclipse 0m 41s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 26s the patch passed +1 javadoc 1m 50s the patch passed +1 unit 8m 42s hadoop-common in the patch passed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. -1 unit 57m 29s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 120m 30s Reason Tests Failed junit tests hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10368 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837172/HDFS-10368-00.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux c62ed277614b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0aafc12 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17433/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17433/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17433/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 40 new or modified test files.
        0 mvndep 1m 40s Maven dependency ordering for branch
        +1 mvninstall 7m 54s trunk passed
        +1 compile 8m 21s trunk passed
        +1 checkstyle 1m 48s trunk passed
        +1 mvnsite 2m 32s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 4m 36s trunk passed
        +1 javadoc 1m 43s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 52s the patch passed
        +1 compile 7m 16s the patch passed
        +1 javac 7m 16s the patch passed
        -0 checkstyle 1m 49s root: The patch generated 1 new + 1624 unchanged - 9 fixed = 1625 total (was 1633)
        +1 mvnsite 2m 24s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 4m 53s the patch passed
        +1 javadoc 1m 40s the patch passed
        +1 unit 7m 39s hadoop-common in the patch passed.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
        -1 unit 55m 23s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 24s The patch does not generate ASF License warnings.
        115m 50s



        Reason Tests
        Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-10368
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837406/HDFS-10368-01.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux c0bd5adcb852 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d8bab3d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17441/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17441/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17441/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17441/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 40 new or modified test files. 0 mvndep 1m 40s Maven dependency ordering for branch +1 mvninstall 7m 54s trunk passed +1 compile 8m 21s trunk passed +1 checkstyle 1m 48s trunk passed +1 mvnsite 2m 32s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 4m 36s trunk passed +1 javadoc 1m 43s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 7m 16s the patch passed +1 javac 7m 16s the patch passed -0 checkstyle 1m 49s root: The patch generated 1 new + 1624 unchanged - 9 fixed = 1625 total (was 1633) +1 mvnsite 2m 24s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 53s the patch passed +1 javadoc 1m 40s the patch passed +1 unit 7m 39s hadoop-common in the patch passed. +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 55m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 115m 50s Reason Tests Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10368 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837406/HDFS-10368-01.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux c0bd5adcb852 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8bab3d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17441/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17441/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17441/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17441/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Hi Andrew Wang, I've tried an attempt to deprecate the keys. Could you please review when you get a chance. Thanks!

        Show
        rakeshr Rakesh R added a comment - Hi Andrew Wang , I've tried an attempt to deprecate the keys. Could you please review when you get a chance. Thanks!
        Hide
        andrew.wang Andrew Wang added a comment -

        Looks good overall, thanks for working on this Rakesh! A few review comments:

        • Rename the name in LOG print in BlockManager line 496?
        • Since we're renaming redundancyRecheckInterval anyway, this is a good opportunity to put some units into the name. ReplicationMonitor can also sleep using TimeUnit#sleep for clarity.
        • Do you think we should rename ReplicationMonitor in BlockManager to RedundancyMonitor?
        • Unused variable alpha in TestDeprecatedKeys. We should also consider checking changed deprecations like dfs.replication.interval as well.
        Show
        andrew.wang Andrew Wang added a comment - Looks good overall, thanks for working on this Rakesh! A few review comments: Rename the name in LOG print in BlockManager line 496? Since we're renaming redundancyRecheckInterval anyway, this is a good opportunity to put some units into the name. ReplicationMonitor can also sleep using TimeUnit#sleep for clarity. Do you think we should rename ReplicationMonitor in BlockManager to RedundancyMonitor? Unused variable alpha in TestDeprecatedKeys. We should also consider checking changed deprecations like dfs.replication.interval as well.
        Hide
        rakeshr Rakesh R added a comment - - edited

        Thank you Andrew Wang for the useful comments. I'll take care in my next patch.

        ReplicationMonitor can also sleep using TimeUnit#sleep for clarity.

        Sorry, I failed to understand this comment, please help me. Thanks!
        Are you suggesting to rename var BlockManager#redundancyRecheckInterval to BlockManager#redundancyRecheckIntervalMillis and use it?

        Show
        rakeshr Rakesh R added a comment - - edited Thank you Andrew Wang for the useful comments. I'll take care in my next patch. ReplicationMonitor can also sleep using TimeUnit#sleep for clarity. Sorry, I failed to understand this comment, please help me. Thanks! Are you suggesting to rename var BlockManager#redundancyRecheckInterval to BlockManager#redundancyRecheckIntervalMillis and use it?
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Rakesh, right now there's this line in ReplicationMonitor:

                  Thread.sleep(replicationRecheckInterval);
        

        I was suggesting that we could use TimeUnit.MILLISECONDS.sleep instead for additional clarity. Up to you though.

        Show
        andrew.wang Andrew Wang added a comment - Hi Rakesh, right now there's this line in ReplicationMonitor: Thread .sleep(replicationRecheckInterval); I was suggesting that we could use TimeUnit.MILLISECONDS.sleep instead for additional clarity. Up to you though.
        Hide
        rakeshr Rakesh R added a comment -

        Thank you. Used TimeUnit.MILLISECONDS.sleep.

        Show
        rakeshr Rakesh R added a comment - Thank you. Used TimeUnit.MILLISECONDS.sleep .
        Hide
        rakeshr Rakesh R added a comment -

        Attached new patch addressing Andrew Wang's comments.

        Show
        rakeshr Rakesh R added a comment - Attached new patch addressing Andrew Wang 's comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 42 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 6m 41s trunk passed
        +1 compile 9m 34s trunk passed
        +1 checkstyle 1m 55s trunk passed
        +1 mvnsite 2m 38s trunk passed
        +1 mvneclipse 0m 52s trunk passed
        +1 findbugs 4m 58s trunk passed
        +1 javadoc 2m 1s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 2m 0s the patch passed
        +1 compile 9m 45s the patch passed
        +1 javac 9m 45s the patch passed
        -0 checkstyle 1m 54s root: The patch generated 19 new + 1810 unchanged - 15 fixed = 1829 total (was 1825)
        +1 mvnsite 2m 34s the patch passed
        +1 mvneclipse 0m 54s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 5m 15s the patch passed
        +1 javadoc 2m 5s the patch passed
        +1 unit 8m 4s hadoop-common in the patch passed.
        +1 unit 1m 3s hadoop-hdfs-client in the patch passed.
        -1 unit 58m 41s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 35s The patch does not generate ASF License warnings.
        123m 36s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.ha.TestHASafeMode
          hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
          hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
          hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
          hadoop.hdfs.TestSetTimes
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10368
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839107/HDFS-10368-02.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux 7dfd25b30e5c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 61c0bed
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17582/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17582/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 42 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 9m 34s trunk passed +1 checkstyle 1m 55s trunk passed +1 mvnsite 2m 38s trunk passed +1 mvneclipse 0m 52s trunk passed +1 findbugs 4m 58s trunk passed +1 javadoc 2m 1s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 0s the patch passed +1 compile 9m 45s the patch passed +1 javac 9m 45s the patch passed -0 checkstyle 1m 54s root: The patch generated 19 new + 1810 unchanged - 15 fixed = 1829 total (was 1825) +1 mvnsite 2m 34s the patch passed +1 mvneclipse 0m 54s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 15s the patch passed +1 javadoc 2m 5s the patch passed +1 unit 8m 4s hadoop-common in the patch passed. +1 unit 1m 3s hadoop-hdfs-client in the patch passed. -1 unit 58m 41s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 123m 36s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestHASafeMode   hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport   hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage   hadoop.hdfs.TestSetTimes   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10368 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839107/HDFS-10368-02.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux 7dfd25b30e5c 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61c0bed Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17582/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17582/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17582/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Attached another patch fixing test case failures and style warnings.

        Show
        rakeshr Rakesh R added a comment - Attached another patch fixing test case failures and style warnings.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 42 new or modified test files.
        0 mvndep 1m 35s Maven dependency ordering for branch
        +1 mvninstall 6m 42s trunk passed
        +1 compile 9m 31s trunk passed
        +1 checkstyle 1m 55s trunk passed
        +1 mvnsite 2m 39s trunk passed
        +1 mvneclipse 0m 54s trunk passed
        +1 findbugs 4m 40s trunk passed
        +1 javadoc 1m 55s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 52s the patch passed
        +1 compile 9m 4s the patch passed
        +1 javac 9m 4s the patch passed
        -0 checkstyle 1m 56s root: The patch generated 2 new + 1808 unchanged - 16 fixed = 1810 total (was 1824)
        +1 mvnsite 2m 33s the patch passed
        +1 mvneclipse 0m 55s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 5m 4s the patch passed
        +1 javadoc 1m 55s the patch passed
        +1 unit 8m 20s hadoop-common in the patch passed.
        +1 unit 1m 1s hadoop-hdfs-client in the patch passed.
        +1 unit 53m 37s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 33s The patch does not generate ASF License warnings.
        118m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10368
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839128/HDFS-10368-03.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux efb4132d42f0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 61c0bed
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17585/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17585/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17585/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 42 new or modified test files. 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 9m 31s trunk passed +1 checkstyle 1m 55s trunk passed +1 mvnsite 2m 39s trunk passed +1 mvneclipse 0m 54s trunk passed +1 findbugs 4m 40s trunk passed +1 javadoc 1m 55s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 9m 4s the patch passed +1 javac 9m 4s the patch passed -0 checkstyle 1m 56s root: The patch generated 2 new + 1808 unchanged - 16 fixed = 1810 total (was 1824) +1 mvnsite 2m 33s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 4s the patch passed +1 javadoc 1m 55s the patch passed +1 unit 8m 20s hadoop-common in the patch passed. +1 unit 1m 1s hadoop-hdfs-client in the patch passed. +1 unit 53m 37s hadoop-hdfs in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 118m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10368 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839128/HDFS-10368-03.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux efb4132d42f0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61c0bed Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17585/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17585/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17585/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rakeshr Rakesh R added a comment -

        Andrew Wang, I've got a better jenkins report. Please review this when you get a chance. Thanks!

        Show
        rakeshr Rakesh R added a comment - Andrew Wang , I've got a better jenkins report. Please review this when you get a chance. Thanks!
        Hide
        andrew.wang Andrew Wang added a comment -

        Sorry for leaving this sitting for so long, looks like the patch no longer applies cleanly. I'm +1 pending a fresh rebase though, thanks again for working on this Rakesh!

        Show
        andrew.wang Andrew Wang added a comment - Sorry for leaving this sitting for so long, looks like the patch no longer applies cleanly. I'm +1 pending a fresh rebase though, thanks again for working on this Rakesh!
        Hide
        rakeshr Rakesh R added a comment -

        Thanks Andrew Wang, I've rebased the previous patch on top of the latest trunk code and attached the same to jira.

        Show
        rakeshr Rakesh R added a comment - Thanks Andrew Wang , I've rebased the previous patch on top of the latest trunk code and attached the same to jira.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 42 new or modified test files.
        0 mvndep 1m 42s Maven dependency ordering for branch
        +1 mvninstall 9m 12s trunk passed
        +1 compile 11m 11s trunk passed
        +1 checkstyle 1m 53s trunk passed
        +1 mvnsite 2m 45s trunk passed
        +1 mvneclipse 0m 56s trunk passed
        +1 findbugs 4m 48s trunk passed
        +1 javadoc 1m 58s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 53s the patch passed
        +1 compile 9m 14s the patch passed
        +1 javac 9m 14s the patch passed
        -0 checkstyle 2m 1s root: The patch generated 2 new + 1714 unchanged - 16 fixed = 1716 total (was 1730)
        +1 mvnsite 2m 38s the patch passed
        +1 mvneclipse 0m 55s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 6m 7s the patch passed
        +1 javadoc 1m 59s the patch passed
        +1 unit 9m 24s hadoop-common in the patch passed.
        +1 unit 1m 2s hadoop-hdfs-client in the patch passed.
        +1 unit 66m 3s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 36s The patch does not generate ASF License warnings.
        138m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-10368
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840179/HDFS-10368-04.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux 2d7d67bb79f0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2bf9a15
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17643/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17643/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17643/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 42 new or modified test files. 0 mvndep 1m 42s Maven dependency ordering for branch +1 mvninstall 9m 12s trunk passed +1 compile 11m 11s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 2m 45s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 4m 48s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 9m 14s the patch passed +1 javac 9m 14s the patch passed -0 checkstyle 2m 1s root: The patch generated 2 new + 1714 unchanged - 16 fixed = 1716 total (was 1730) +1 mvnsite 2m 38s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 6m 7s the patch passed +1 javadoc 1m 59s the patch passed +1 unit 9m 24s hadoop-common in the patch passed. +1 unit 1m 2s hadoop-hdfs-client in the patch passed. +1 unit 66m 3s hadoop-hdfs in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 138m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10368 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840179/HDFS-10368-04.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux 2d7d67bb79f0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2bf9a15 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17643/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17643/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17643/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks for working on this Rakesh!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for working on this Rakesh!
        Hide
        rakeshr Rakesh R added a comment -

        Andrew Wang, thank you for the helpful code review feedback and the commit!

        Show
        rakeshr Rakesh R added a comment - Andrew Wang , thank you for the helpful code review feedback and the commit!

          People

          • Assignee:
            rakeshr Rakesh R
            Reporter:
            rakeshr Rakesh R
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development