Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 1.0.0
    • Fix Version/s: 0.22.0, 1.2.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There should be some documentation for HFTP.

      1. HDFS-1957.patch
        4 kB
        Ari Rabkin
      2. HDFS-1957.patch
        4 kB
        Ari Rabkin
      3. HDFS-1957.patch
        3 kB
        Ari Rabkin

        Activity

        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop 1.2.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
        Hide
        Eli Collins added a comment -

        I merged this to branch-1. Verified the generated hftp.html page is sane and linked into the index correctly.

        Show
        Eli Collins added a comment - I merged this to branch-1. Verified the generated hftp.html page is sane and linked into the index correctly.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #53 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/53/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #53 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/53/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #677 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/677/ )
        Hide
        Todd Lipcon added a comment -

        Committed to 22 and trunk. Thanks, Ari!

        Show
        Todd Lipcon added a comment - Committed to 22 and trunk. Thanks, Ari!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > ... Is the current text a bad way to say that?

        The current text is good. I misread it earlier.

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - > ... Is the current text a bad way to say that? The current text is good. I misread it earlier. +1 patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479719/HDFS-1957.patch
        against trunk revision 1124459.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479719/HDFS-1957.patch against trunk revision 1124459. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/580//console This message is automatically generated.
        Hide
        Ari Rabkin added a comment -

        d'oh. The dangers of programming by cut+paste. I think that the label there doesn't affect anything user visible, which is how it slipped by, but here is a revised version.

        Show
        Ari Rabkin added a comment - d'oh. The dangers of programming by cut+paste. I think that the label there doesn't affect anything user visible, which is how it slipped by, but here is a revised version.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479705/HDFS-1957.patch
        against trunk revision 1124459.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479705/HDFS-1957.patch against trunk revision 1124459. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/576//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        This looks suspect:
        + <hdfsproxy label="HFTP" href="hftp.html"/>
        shouldn't this say <hftp> or something?

        Show
        Todd Lipcon added a comment - This looks suspect: + <hdfsproxy label="HFTP" href="hftp.html"/> shouldn't this say <hftp> or something?
        Hide
        Ari Rabkin added a comment -

        Revised as per comments from Todd and Nicholas.

        Show
        Ari Rabkin added a comment - Revised as per comments from Todd and Nicholas.
        Hide
        Ari Rabkin added a comment -

        If I read the code right, there is a distinct option, of the form hdfs.service.host_xyz, for each ip:port pair xyz. Is the current text a bad way to say that?

        Show
        Ari Rabkin added a comment - If I read the code right, there is a distinct option, of the form hdfs.service.host_xyz, for each ip:port pair xyz. Is the current text a bad way to say that?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479673/HDFS-1957.patch
        against trunk revision 1124364.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
        org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.tools.TestJMXGet

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479673/HDFS-1957.patch against trunk revision 1124364. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.tools.TestJMXGet +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/567//console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Is the "_<strong>ip:port</strong>" not a part of the conf property name?

        Show
        Tsz Wo Nicholas Sze added a comment - Is the "_<strong>ip:port</strong>" not a part of the conf property name?
        Hide
        Ari Rabkin added a comment -

        I don't see what typo you're referring to.

        I will revise to mention HSFTP and reference the implementation classes.

        Show
        Ari Rabkin added a comment - I don't see what typo you're referring to. I will revise to mention HSFTP and reference the implementation classes.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • Typo:
          +          <td>hdfs.service.host_<strong>ip:port</strong></td>
          
        • More precisely, it will throw exceptions if you try to modify the filesystem state. For example, creating, deleting or renaming a file/directory.
          +      HFTP is a read-only filesystem, and will throw exceptions if you try to use it to write data.</p>
          
        • Suggestion: you may want to mention HSFTP.
        • Suggestion: you may want to mention the java classes: HftpFileSystem and HsftpFileSystem.
        • Suggestion: it would be easier for reviewing if you also post the generated doc.

        Thanks for working on this.

        Show
        Tsz Wo Nicholas Sze added a comment - Typo: + <td>hdfs.service.host_<strong>ip:port</strong></td> More precisely, it will throw exceptions if you try to modify the filesystem state. For example, creating, deleting or renaming a file/directory. + HFTP is a read-only filesystem, and will throw exceptions if you try to use it to write data.</p> Suggestion: you may want to mention HSFTP. Suggestion: you may want to mention the java classes: HftpFileSystem and HsftpFileSystem . Suggestion: it would be easier for reviewing if you also post the generated doc. Thanks for working on this.
        Hide
        Todd Lipcon added a comment -

        Thanks for writing this! A few comments:

        The reads are done via HTTPS,

        I believe it only uses HTTP by default.

        <p>HFTP is primarily useful if you have multiple HDFS clusters and you need to move data from one to another.</p>

        Should point out its most common use case: HFTP is wire-compatible even between different versions of HDFS - eg you can copy from a previous major release to a cluster running a new one, which is not true of the hdfs:// protocol.

        If possible, it would be nice to add a sample command line using distcp with hftp:// as the source and hdfs:// as the destination. I bet if you search the mailing list you can find a good example command line.

        Show
        Todd Lipcon added a comment - Thanks for writing this! A few comments: The reads are done via HTTPS, I believe it only uses HTTP by default. <p>HFTP is primarily useful if you have multiple HDFS clusters and you need to move data from one to another.</p> Should point out its most common use case: HFTP is wire-compatible even between different versions of HDFS - eg you can copy from a previous major release to a cluster running a new one, which is not true of the hdfs:// protocol. If possible, it would be nice to add a sample command line using distcp with hftp:// as the source and hdfs:// as the destination. I bet if you search the mailing list you can find a good example command line.
        Hide
        Ari Rabkin added a comment -

        At least a partial fix. All docs, no tests.

        Show
        Ari Rabkin added a comment - At least a partial fix. All docs, no tests.

          People

          • Assignee:
            Ari Rabkin
            Reporter:
            Ari Rabkin
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development