Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      There are a couple issues with the current distcp document:

      • It recommends hftp / hsftp filesystem to copy data between different hadoop version. hftp / hsftp have been deprecated in the flavor of webhdfs.
      • If the users are copying between Hadoop 2.x they can use the hdfs protocol directly for better performance.
      1. HDFS-9048-4.patch
        1 kB
        Daisuke Kobayashi
      2. HDFS-9048-3.patch
        1 kB
        Daisuke Kobayashi
      3. HDFS-9048-2.patch
        1 kB
        Daisuke Kobayashi
      4. HDFS-9048.patch
        1 kB
        Daisuke Kobayashi

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9418 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9418/)
        HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via (iwasakims: rev 33a412e8a4ab729d588a9576fb7eb90239c6e383)

        • hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9418 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9418/ ) HDFS-9048 . DistCp documentation is out-of-dated (Daisuke Kobayashi via (iwasakims: rev 33a412e8a4ab729d588a9576fb7eb90239c6e383) hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        iwasakims Masatake Iwasaki added a comment -

        +1. Committed to branch-2.7 and above. Thanks, Daisuke Kobayashi!

        Show
        iwasakims Masatake Iwasaki added a comment - +1. Committed to branch-2.7 and above. Thanks, Daisuke Kobayashi !
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 48s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvnsite 0m 20s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        8m 19s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791130/HDFS-9048-4.patch
        JIRA Issue HDFS-9048
        Optional Tests asflicense mvnsite
        uname Linux 03df14ace075 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f70e317
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14702/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 48s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 8m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791130/HDFS-9048-4.patch JIRA Issue HDFS-9048 Optional Tests asflicense mvnsite uname Linux 03df14ace075 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f70e317 modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14702/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daisuke.kobayashi Daisuke Kobayashi added a comment -

        Uploaded a v4 patch. Thanks.

        Show
        daisuke.kobayashi Daisuke Kobayashi added a comment - Uploaded a v4 patch. Thanks.
        Hide
        iwasakims Masatake Iwasaki added a comment -

        namenode_address or namenode_host should be fine.

        Show
        iwasakims Masatake Iwasaki added a comment - namenode_address or namenode_host should be fine.
        Hide
        daisuke.kobayashi Daisuke Kobayashi added a comment -

        Ah, thanks for pointing me out to the right direction, Masatake Iwasaki. So for easier understanding for readers, should I be webhdfs://<namenode_address>:<http_port>? Thoughts?

        Show
        daisuke.kobayashi Daisuke Kobayashi added a comment - Ah, thanks for pointing me out to the right direction, Masatake Iwasaki . So for easier understanding for readers, should I be webhdfs://<namenode_address>:<http_port> ? Thoughts?
        Hide
        iwasakims Masatake Iwasaki added a comment -

        Thanks for the update Daisuke Kobayashi. If you prefer using configuration properties, webhdfs://<dfs.http.address>:<http_port> should be webhdfs://<dfs.http.address> since the value of dfs.http.address (and dfs.namenode.http-address which is the successor) includes port number.

        I'm +1 if this is addressed.

        Show
        iwasakims Masatake Iwasaki added a comment - Thanks for the update Daisuke Kobayashi . If you prefer using configuration properties, webhdfs://<dfs.http.address>:<http_port> should be webhdfs://<dfs.http.address> since the value of dfs.http.address (and dfs.namenode.http-address which is the successor) includes port number. I'm +1 if this is addressed.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 6m 48s trunk passed
        +1 mvnsite 0m 23s trunk passed
        +1 mvnsite 0m 20s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        8m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790906/HDFS-9048-3.patch
        JIRA Issue HDFS-9048
        Optional Tests asflicense mvnsite
        uname Linux d73af39a392e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 27e0681
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14686/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 48s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 8m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790906/HDFS-9048-3.patch JIRA Issue HDFS-9048 Optional Tests asflicense mvnsite uname Linux d73af39a392e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27e0681 modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14686/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daisuke.kobayashi Daisuke Kobayashi added a comment -

        Masatake Iwasaki - yup, that's reasonable! Updated per your suggestion.

        Show
        daisuke.kobayashi Daisuke Kobayashi added a comment - Masatake Iwasaki - yup, that's reasonable! Updated per your suggestion.
        Hide
        iwasakims Masatake Iwasaki added a comment -
        419	  Remote cluster is specified as `webhdfs://<dfs.http.address>/<path>`
        420	  (the default `dfs.http.address` is `<namenode>:50070`). 
        

        dfs.http.address is deplicated and the default are possible to be changed by HDFS-9427. How about just saying "Remote cluster is specified as 'webhdfs://<namenode>:<http_port>/<path>'"?

        Show
        iwasakims Masatake Iwasaki added a comment - 419 Remote cluster is specified as `webhdfs://<dfs.http.address>/<path>` 420 (the default `dfs.http.address` is `<namenode>:50070`). dfs.http.address is deplicated and the default are possible to be changed by HDFS-9427 . How about just saying "Remote cluster is specified as 'webhdfs://<namenode>:<http_port>/<path>'"?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 12m 59s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvnsite 0m 43s trunk passed
        +1 mvnsite 0m 20s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        14m 46s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788603/HDFS-9048-2.patch
        JIRA Issue HDFS-9048
        Optional Tests asflicense mvnsite
        uname Linux de56f9d15704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 321a80c
        modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14647/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 12m 59s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 43s trunk passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 14m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788603/HDFS-9048-2.patch JIRA Issue HDFS-9048 Optional Tests asflicense mvnsite uname Linux de56f9d15704 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 321a80c modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14647/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daisuke.kobayashi Daisuke Kobayashi added a comment -

        Thanks Wei-Chiu Chuang! Uploaded a new one. Yea, committing in 2.x would be fine.

        Show
        daisuke.kobayashi Daisuke Kobayashi added a comment - Thanks Wei-Chiu Chuang ! Uploaded a new one. Yea, committing in 2.x would be fine.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I think it's fine to commit this in 2.x. webhdfs is preferred than hftp even in 2.x.

        However, webhdfs, unlike hftp, supports both read and write. So distcp does not need to run on destination cluster.

        Show
        jojochuang Wei-Chiu Chuang added a comment - I think it's fine to commit this in 2.x. webhdfs is preferred than hftp even in 2.x. However, webhdfs, unlike hftp, supports both read and write. So distcp does not need to run on destination cluster.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Daisuke Kobayashi Thanks for working on this. The patch looks good to me.
        What's the target version of this patch?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Daisuke Kobayashi Thanks for working on this. The patch looks good to me. What's the target version of this patch?
        Hide
        daisuke.kobayashi Daisuke Kobayashi added a comment -

        Assigned this to me and attached a patch. Haohui Mai, can you review this please?

        Show
        daisuke.kobayashi Daisuke Kobayashi added a comment - Assigned this to me and attached a patch. Haohui Mai , can you review this please?

          People

          • Assignee:
            daisuke.kobayashi Daisuke Kobayashi
            Reporter:
            wheat9 Haohui Mai
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development