Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-472

Document hdfsproxy design and set-up guide

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/hdfsproxy
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      currently hdfsproxy only have a README file that does not follow closely to the code. Need more documentation on the design, build and set-up guide.

      1. HDFS-472.patch
        182 kB
        zhiyong zhang
      2. HDFS-472.patch
        176 kB
        zhiyong zhang
      3. HDFS-472.patch
        176 kB
        zhiyong zhang
      4. hdfsproxy.pdf
        192 kB
        zhiyong zhang
      5. hdfsproxy.pdf
        151 kB
        zhiyong zhang

        Activity

        Hide
        zhiyong zhang added a comment -

        This PDF file is generated by forrest. It's easier to view. After collecting some feedback, I will upload the forrest patch.
        Thanks.

        Show
        zhiyong zhang added a comment - This PDF file is generated by forrest. It's easier to view. After collecting some feedback, I will upload the forrest patch. Thanks.
        Hide
        zhiyong zhang added a comment -

        1. revised some sentences according to kan's comments.
        2. added client-configuration part.

        Show
        zhiyong zhang added a comment - 1. revised some sentences according to kan's comments. 2. added client-configuration part.
        Hide
        zhiyong zhang added a comment -

        forrest patch. Including 4 .jpg diagrams.

        Show
        zhiyong zhang added a comment - forrest patch. Including 4 .jpg diagrams.
        Hide
        Chris Douglas added a comment -

        Thanks for writing such thorough documentation. A few nits:

        • The "advantages" and "disadvantages" in the comparison w/ H(S)FTP could be formatted more clearly in a list or table.
        • 6.6.1 and 6.6.2 are missing a line break before the "ant clean tar" shell command
        • There are several occurrences of "Ip address" (e.g. 4.2.2 sub 3) that should read "IP address"
        • Why are dfs.block.size and io.file.buffer.size important parameters for the proxy?
        • There are three "hdfsproxy-default.xml" tables and all tables are labeled "Table 1" (the structure is much clearer in the HTML rendering than in the PDF; looks like a forrest quirk)
        • Has someone unfamiliar with the details of the project successfully configured an instance of HDFS proxy from this documentation?
        Show
        Chris Douglas added a comment - Thanks for writing such thorough documentation. A few nits: The "advantages" and "disadvantages" in the comparison w/ H(S)FTP could be formatted more clearly in a list or table. 6.6.1 and 6.6.2 are missing a line break before the "ant clean tar" shell command There are several occurrences of "Ip address" (e.g. 4.2.2 sub 3) that should read "IP address" Why are dfs.block.size and io.file.buffer.size important parameters for the proxy? There are three "hdfsproxy-default.xml" tables and all tables are labeled "Table 1" (the structure is much clearer in the HTML rendering than in the PDF; looks like a forrest quirk) Has someone unfamiliar with the details of the project successfully configured an instance of HDFS proxy from this documentation?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12414645/HDFS-472.patch
        against trunk revision 806746.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414645/HDFS-472.patch against trunk revision 806746. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/83/console This message is automatically generated.
        Hide
        zhiyong zhang added a comment -

        Hi Chris,
        Thanks for the comments. I uploaded the new patch.

        • The "advantages" and "disadvantages" in the comparison w/ H(S)FTP could be formatted more clearly in a list or table.
          --> yes. I listed them.
        • 6.6.1 and 6.6.2 are missing a line break before the "ant clean tar" shell command
          --> good catch. I corrected a couple more.
        • There are several occurrences of "Ip address" (e.g. 4.2.2 sub 3) that should read "IP address"
          --> changed
        • Why are dfs.block.size and io.file.buffer.size important parameters for the proxy?
          --> They are not crucial to the functionality of proxy. Fine tuning these parameters could affect the transfer performance to some degree. This should have been documented in hdfs-client.
        • There are three "hdfsproxy-default.xml" tables and all tables are labeled "Table 1" (the structure is much clearer in the HTML rendering than in the PDF; looks like a forrest quirk)
          --> seems <caption> tag inside <table> block is causing the problem. Replaced it with <strong> out of the <table> block. Now all look the same in HTML and PDF.
        • Has someone unfamiliar with the details of the project successfully configured an instance of HDFS proxy from this documentation?
          --> I did that. But maybe not fair since I am relatively familiar with it.
        Show
        zhiyong zhang added a comment - Hi Chris, Thanks for the comments. I uploaded the new patch. The "advantages" and "disadvantages" in the comparison w/ H(S)FTP could be formatted more clearly in a list or table. --> yes. I listed them. 6.6.1 and 6.6.2 are missing a line break before the "ant clean tar" shell command --> good catch. I corrected a couple more. There are several occurrences of "Ip address" (e.g. 4.2.2 sub 3) that should read "IP address" --> changed Why are dfs.block.size and io.file.buffer.size important parameters for the proxy? --> They are not crucial to the functionality of proxy. Fine tuning these parameters could affect the transfer performance to some degree. This should have been documented in hdfs-client. There are three "hdfsproxy-default.xml" tables and all tables are labeled "Table 1" (the structure is much clearer in the HTML rendering than in the PDF; looks like a forrest quirk) --> seems <caption> tag inside <table> block is causing the problem. Replaced it with <strong> out of the <table> block. Now all look the same in HTML and PDF. Has someone unfamiliar with the details of the project successfully configured an instance of HDFS proxy from this documentation? --> I did that. But maybe not fair since I am relatively familiar with it.
        Hide
        Chris Douglas added a comment -

        +1 The latest version looks good.

        Would it make sense to update the README as part of this issue? Even if it only briefly summarizes what the project does and refers the reader to the forrest documentation, that would likely be more helpful than docs for a previous version of the proxy.

        Show
        Chris Douglas added a comment - +1 The latest version looks good. Would it make sense to update the README as part of this issue? Even if it only briefly summarizes what the project does and refers the reader to the forrest documentation, that would likely be more helpful than docs for a previous version of the proxy.
        Hide
        zhiyong zhang added a comment -

        changed README file

        Show
        zhiyong zhang added a comment - changed README file
        Hide
        Chris Douglas added a comment -

        (only change is to README, so the previous doc +1 from Hudson should be good)

        I committed this. Thanks, Zhiyong!

        Show
        Chris Douglas added a comment - (only change is to README, so the previous doc +1 from Hudson should be good) I committed this. Thanks, Zhiyong!
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/)

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/ )
        Hide
        Hudson added a comment -

        Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/)
        . Update hdfsproxy documentation. Adds a setup guide and design
        document. Contributed by Zhiyong Zhang

        Show
        Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/ ) . Update hdfsproxy documentation. Adds a setup guide and design document. Contributed by Zhiyong Zhang
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #34 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/34/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #34 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/34/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #84 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/84/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #84 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/84/ )

          People

          • Assignee:
            zhiyong zhang
            Reporter:
            zhiyong zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development