Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13587

distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: tools/distcp
    • Labels:
      None

      Description

      distcp.map.bandwidth.mb exists in distcp-defaults.xml config file, but it is not honored even when it is . Current code always overwrites it with either default value (java const) or with -bandwidth command line option.

      The expected behavior (at least how I would expect it) is to honor the value set in distcp-defaults.xml unless user explicitly specify -bandwidth command line flag. If there is no value set in .xml file or as a command line flag, then the constant from java code should be used.

      Additionally, I would expect that we also try to get values from distcp-site.xml, similar to other hadoop systems.

      1. HADOOP-13587-01.patch
        8 kB
        Zoran Dimitrijevic
      2. HADOOP-13587-02.patch
        8 kB
        Zoran Dimitrijevic

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10425/)
        HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when (raviprak: rev 9faccd104672dfef123735ca8ada178fc3a6196f)

        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
        • (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
        • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10425 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10425/ ) HADOOP-13587 . distcp.map.bandwidth.mb is overwritten even when (raviprak: rev 9faccd104672dfef123735ca8ada178fc3a6196f) (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
        Hide
        raviprak Ravi Prakash added a comment -

        Committed to trunk! Thanks for the contribution Zoran!

        Show
        raviprak Ravi Prakash added a comment - Committed to trunk! Thanks for the contribution Zoran!
        Hide
        raviprak Ravi Prakash added a comment -

        +1. LGTM. Thanks a lot for your contribution Zoran. Committing shortly!

        Show
        raviprak Ravi Prakash added a comment - +1. LGTM. Thanks a lot for your contribution Zoran. Committing shortly!
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 13s trunk passed
        +1 compile 0m 17s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        +1 checkstyle 0m 11s hadoop-tools/hadoop-distcp: The patch generated 0 new + 81 unchanged - 1 fixed = 81 total (was 82)
        +1 mvnsite 0m 17s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 28s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 9m 37s hadoop-distcp in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        22m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13587
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827645/HADOOP-13587-02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d78ccbd7f4e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b6d839a
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10464/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10464/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 13s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-distcp: The patch generated 0 new + 81 unchanged - 1 fixed = 81 total (was 82) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 9m 37s hadoop-distcp in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 22m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13587 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827645/HADOOP-13587-02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d78ccbd7f4e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6d839a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10464/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10464/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 3s trunk passed
        +1 compile 0m 17s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 20s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 24s trunk passed
        +1 javadoc 0m 12s trunk passed
        +1 mvninstall 0m 15s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        -0 checkstyle 0m 11s hadoop-tools/hadoop-distcp: The patch generated 10 new + 81 unchanged - 1 fixed = 91 total (was 82)
        +1 mvnsite 0m 17s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 31s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 9m 29s hadoop-distcp in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        21m 50s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HADOOP-13587
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827451/HADOOP-13587-01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 174b636187c7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f414d5e
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/testReport/
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-distcp: The patch generated 10 new + 81 unchanged - 1 fixed = 91 total (was 82) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 31s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 9m 29s hadoop-distcp in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 21m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13587 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827451/HADOOP-13587-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 174b636187c7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f414d5e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10459/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            3opan Zoran Dimitrijevic
            Reporter:
            3opan Zoran Dimitrijevic
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development