Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: documentation, scripts
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Python is now required to build the documentation.

      Description

      After HADOOP-11485, we need to formally document functions and environment variables that 3rd parties can expect to be able to exist/use.

      1. HADOOP-11553-00.patch
        69 kB
        Allen Wittenauer
      2. HADOOP-11553-01.patch
        73 kB
        Allen Wittenauer
      3. HADOOP-11553-02.patch
        31 kB
        Allen Wittenauer
      4. HADOOP-11553-03.patch
        38 kB
        Allen Wittenauer
      5. HADOOP-11553-04.patch
        41 kB
        Allen Wittenauer
      6. HADOOP-11553-05.patch
        41 kB
        Allen Wittenauer
      7. HADOOP-11553-06.patch
        41 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/136/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-project/src/site/site.xml
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • dev-support/shelldocs.py
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #136 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/136/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-project/src/site/site.xml hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh dev-support/shelldocs.py HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2077 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2077/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-project/src/site/site.xml
          • dev-support/shelldocs.py
          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/pom.xml
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2077 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2077/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-project/src/site/site.xml dev-support/shelldocs.py hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/pom.xml HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2095 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2095/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-project/src/site/site.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • dev-support/shelldocs.py
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2095 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2095/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-project/src/site/site.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md dev-support/shelldocs.py HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #145 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/145/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/shelldocs.py
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-project/src/site/site.xml
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #145 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/145/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt dev-support/shelldocs.py hadoop-common-project/hadoop-common/pom.xml hadoop-project/src/site/site.xml HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #879 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/879/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-project/src/site/site.xml
          • hadoop-common-project/hadoop-common/pom.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • dev-support/shelldocs.py
          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #879 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/879/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-project/src/site/site.xml hadoop-common-project/hadoop-common/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt dev-support/shelldocs.py hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #145 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/145/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/pom.xml
          • dev-support/shelldocs.py
          • hadoop-project/src/site/site.xml
          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #145 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/145/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/pom.xml dev-support/shelldocs.py hadoop-project/src/site/site.xml hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/CHANGES.txt HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7443 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7443/)
          HADOOP-11553. Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793)

          • hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-project/src/site/site.xml
          • dev-support/shelldocs.py
          • hadoop-common-project/hadoop-common/pom.xml
            HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3)
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7443 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7443/ ) HADOOP-11553 . Foramlize the shell API (aw) (aw: rev b30ca8ce0e0d435327e179f0877bd58fa3896793) hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-project/src/site/site.xml dev-support/shelldocs.py hadoop-common-project/hadoop-common/pom.xml HADOOP-11553 addendum fix the typo in the changes file (aw: rev 5695c7a541c1a3092040523446f1ba689fb495e3) hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          aw Allen Wittenauer added a comment -

          This has been committed to trunk.

          Thanks for the review!

          Show
          aw Allen Wittenauer added a comment - This has been committed to trunk. Thanks for the review!
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch v06. Thank you for the documentation, Allen.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch v06. Thank you for the documentation, Allen.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707572/HADOOP-11553-06.patch
          against trunk revision 87130bf.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6004//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6004//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707572/HADOOP-11553-06.patch against trunk revision 87130bf. +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/6004//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/6004//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          -06:

          • Fixed those spelling errors

          Thanks for the reviews, btw.

          Show
          aw Allen Wittenauer added a comment - -06: Fixed those spelling errors Thanks for the reviews, btw.
          Hide
          cnauroth Chris Nauroth added a comment -

          I think we're close. Just a few more typos:

          • should be a colon delimited list of dictories. (Change to "directories".)
          • or wildcard locatons. (Change to "locations".)
          Show
          cnauroth Chris Nauroth added a comment - I think we're close. Just a few more typos: should be a colon delimited list of dictories . (Change to "directories".) or wildcard locatons . (Change to "locations".)
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707011/HADOOP-11553-05.patch
          against trunk revision a16bfff.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5990//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5990//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707011/HADOOP-11553-05.patch against trunk revision a16bfff. +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5990//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5990//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Linking HADOOP-9642. If both get committed, the docs should reference that functionality.

          Show
          aw Allen Wittenauer added a comment - Linking HADOOP-9642 . If both get committed, the docs should reference that functionality.
          Hide
          aw Allen Wittenauer added a comment -

          -05:

          • Address Chris Nauroth's feedback (I think)
          • shelldocs.py: added the extremely obvious --skipprnorep option to prevent private + not replaceable functions from being output'ed
          Show
          aw Allen Wittenauer added a comment - -05: Address Chris Nauroth 's feedback (I think) shelldocs.py: added the extremely obvious --skipprnorep option to prevent private + not replaceable functions from being output'ed
          Hide
          aw Allen Wittenauer added a comment -

          For shelldocmd.py, what Python version did you use in testing?

          I've been using what is on my Mac, which happens to be 2.7.5 I've been trying to avoid using anything too fancy, with 2.5 being the version I'm shooting for.

          My last comment was based on the 03 patch, and I fell just behind of you posting the 04 patch.

          I'll go through your comments to see what I missed in -04.

          Show
          aw Allen Wittenauer added a comment - For shelldocmd.py, what Python version did you use in testing? I've been using what is on my Mac, which happens to be 2.7.5 I've been trying to avoid using anything too fancy, with 2.5 being the version I'm shooting for. My last comment was based on the 03 patch, and I fell just behind of you posting the 04 patch. I'll go through your comments to see what I missed in -04.
          Hide
          cnauroth Chris Nauroth added a comment -

          My last comment was based on the 03 patch, and I fell just behind of you posting the 04 patch.

          Show
          cnauroth Chris Nauroth added a comment - My last comment was based on the 03 patch, and I fell just behind of you posting the 04 patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          Hi Allen. Thank you for putting together this documentation.

          For shelldocmd.py, what Python version did you use in testing? I don't know for sure that it's still the case, but I recall that fairly recent CentOS versions are pinned to Python 2.6 as part of the implementation of yum. I'd like to suggest that we stick to base 2.6 with no additional modules required. (I don't see any red flags so far.)

          UnixShellGuide.md doesn't appear to be hyperlinked from anywhere else in the documentation. Shall we add it to the left nav?

          Shall we omit all private non-replaceable functions from the documentation? On the Java side, we filter out the private things. It's still helpful for maintainers to have the full docs on these functions in the source code, but I don't think we need to publish it to end users.

          hadoop_add_colonpath probably needs more context in the description. We could mention that the default implementations of hadoop_add_javalibpath and hadoop_add_ldlibpath use this. It doesn't currently mention that the first argument is the name of the out variable to receive the modification. Alternatively, I think we could declare this one as a private non-replaceable implementation detail and filter it out of the docs as per above comment.

          Here are a few typos I spotted:

          • is controlled via [the shell](CommandsManula.html) (The hyperlink target should be CommandsManual.html.)
          • HADOOP_CLIENT_OPTS="-Xmx1g -Dhadoop.socks.server=localhost:4000 hadoop fs -ls /tmp" (I think you meant to close the double-quote before the start of the hadoop command.)
          • to run hadoop commands access the server (I think this was supposed to be "accessing".)
          • In the .hadooprc example, the if statement has an extra closing curly brace around HADOOP_SERVER.
          • There are many enironment variables (It should be "environment".)
          • the series of `_OPT` variables (I think this was supposed to be OPTS.)
          • Advanced administrators may which to supplement (This should be "may wish".)
          • provides the capabilities to do funcion overrides (This should be "function".)
          • Different places in the doc say either "run time" or "runtime". Let's use one consistently, probably "runtime".
          • Hadoop's shell code has a [function library](./HadoopShellFunctionAPI.html) (The hyperlink target doesn't match the actual file name, which is UnixShellAPI.html.)
          • Print a message to stderr if –debug is turuned on (This should use "turned on".)
          Show
          cnauroth Chris Nauroth added a comment - Hi Allen. Thank you for putting together this documentation. For shelldocmd.py, what Python version did you use in testing? I don't know for sure that it's still the case, but I recall that fairly recent CentOS versions are pinned to Python 2.6 as part of the implementation of yum. I'd like to suggest that we stick to base 2.6 with no additional modules required. (I don't see any red flags so far.) UnixShellGuide.md doesn't appear to be hyperlinked from anywhere else in the documentation. Shall we add it to the left nav? Shall we omit all private non-replaceable functions from the documentation? On the Java side, we filter out the private things. It's still helpful for maintainers to have the full docs on these functions in the source code, but I don't think we need to publish it to end users. hadoop_add_colonpath probably needs more context in the description. We could mention that the default implementations of hadoop_add_javalibpath and hadoop_add_ldlibpath use this. It doesn't currently mention that the first argument is the name of the out variable to receive the modification. Alternatively, I think we could declare this one as a private non-replaceable implementation detail and filter it out of the docs as per above comment. Here are a few typos I spotted: is controlled via [the shell](CommandsManula.html) (The hyperlink target should be CommandsManual.html.) HADOOP_CLIENT_OPTS="-Xmx1g -Dhadoop.socks.server=localhost:4000 hadoop fs -ls /tmp" (I think you meant to close the double-quote before the start of the hadoop command.) to run hadoop commands access the server (I think this was supposed to be "accessing".) In the .hadooprc example, the if statement has an extra closing curly brace around HADOOP_SERVER. There are many enironment variables (It should be "environment".) the series of `_OPT` variables (I think this was supposed to be OPTS.) Advanced administrators may which to supplement (This should be "may wish".) provides the capabilities to do funcion overrides (This should be "function".) Different places in the doc say either "run time" or "runtime". Let's use one consistently, probably "runtime". Hadoop's shell code has a [function library](./HadoopShellFunctionAPI.html) (The hyperlink target doesn't match the actual file name, which is UnixShellAPI.html.) Print a message to stderr if –debug is turuned on (This should use "turned on".)
          Hide
          aw Allen Wittenauer added a comment - - edited

          -04:

          • toc on the API guide
          • integration into site index
          • minor typo fixed in do_classpath
          • minor cleanup in the shell guide
          • release warning on the shell guide fixed
          Show
          aw Allen Wittenauer added a comment - - edited -04: toc on the API guide integration into site index minor typo fixed in do_classpath minor cleanup in the shell guide release warning on the shell guide fixed
          Hide
          aw Allen Wittenauer added a comment -

          Making HADOOP-11731 as a conflicting patch since they touch the same pom.xml in the same place.

          Show
          aw Allen Wittenauer added a comment - Making HADOOP-11731 as a conflicting patch since they touch the same pom.xml in the same place.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12706185/HADOOP-11553-03.patch
          against trunk revision e1feb4e.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//artifact/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706185/HADOOP-11553-03.patch against trunk revision e1feb4e. +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5979//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          FWIW, I think this is getting very close to commit-able. It'd be good for someone to start looking this over....

          Show
          aw Allen Wittenauer added a comment - FWIW, I think this is getting very close to commit-able. It'd be good for someone to start looking this over....
          Hide
          aw Allen Wittenauer added a comment -

          -03:

          • Integrate shelldocmd.py into the build process.
          • Fix up some minor problems in pom.xml
          • shelldocmd.py now takes options, builds a toc, minor formatting fixes
          • moar dox
          Show
          aw Allen Wittenauer added a comment - -03: Integrate shelldocmd.py into the build process. Fix up some minor problems in pom.xml shelldocmd.py now takes options, builds a toc, minor formatting fixes moar dox
          Hide
          aw Allen Wittenauer added a comment - - edited

          Couple of other things:

          • the md maker obviously needs some header and footer stuff as well as taking a parameter of the shell code to actually process
          • I've uploaded sample output here: https://github.com/aw-altiscale/hadoop-release-metadata/blob/master/3.0.0/shellfunctions.md of the current documentation that is generated. It's not GREAT! but better than what we had.
          • I've (obviously) put the order back on the functions for now. I'm thinking it might be easier to reorder within shelldocmd instead.
          Show
          aw Allen Wittenauer added a comment - - edited Couple of other things: the md maker obviously needs some header and footer stuff as well as taking a parameter of the shell code to actually process I've uploaded sample output here: https://github.com/aw-altiscale/hadoop-release-metadata/blob/master/3.0.0/shellfunctions.md of the current documentation that is generated. It's not GREAT! but better than what we had. I've (obviously) put the order back on the functions for now. I'm thinking it might be easier to reorder within shelldocmd instead.
          Hide
          aw Allen Wittenauer added a comment -

          -02:

          • reformatted documentation in the hadoop-functions.sh
          • shelldocmd.py

          shelldocmd.py is like my first, from scratch python program. It is a javadoc-inspired markdown documentation generator for shell code. I have no idea how to integrate this into Maven however so that we get nice shell function docs.... so will definitely be looking for help there.

          Show
          aw Allen Wittenauer added a comment - -02: reformatted documentation in the hadoop-functions.sh shelldocmd.py shelldocmd.py is like my first, from scratch python program. It is a javadoc-inspired markdown documentation generator for shell code. I have no idea how to integrate this into Maven however so that we get nice shell function docs.... so will definitely be looking for help there.
          Hide
          aw Allen Wittenauer added a comment -

          -01:

          • start of some docs
          Show
          aw Allen Wittenauer added a comment - -01: start of some docs
          Hide
          aw Allen Wittenauer added a comment -

          -00:

          • Initial patch, "code-level" only

          This patch doesn't change any functions but does:

          • puts a stability level, param, etc, comment in front of every function, in a kind of a fake javadoc format
          • re-arranges hadoop-functions to be grouped by those stability levels

          Some things I'm looking for here:

          • Are these the right settings for these functions? Too liberal? Too conservative?
          • Is re-arranging the right thing to do?

          I recognize that the last point isn't something we normally do, breaks patches, etc, etc. However, in the case of the shell code, since there is one big file rather than multiple, I think it would be useful for others reading this file. It forces the most interesting functions closer to the top of a very big file.

          To-do:

          • Write all the docs (function hint guide, shell profile docs, .hadooprc, etc, etc)
          Show
          aw Allen Wittenauer added a comment - -00: Initial patch, "code-level" only This patch doesn't change any functions but does: puts a stability level, param, etc, comment in front of every function, in a kind of a fake javadoc format re-arranges hadoop-functions to be grouped by those stability levels Some things I'm looking for here: Are these the right settings for these functions? Too liberal? Too conservative? Is re-arranging the right thing to do? I recognize that the last point isn't something we normally do, breaks patches, etc, etc. However, in the case of the shell code, since there is one big file rather than multiple, I think it would be useful for others reading this file. It forces the most interesting functions closer to the top of a very big file. To-do: Write all the docs (function hint guide, shell profile docs, .hadooprc, etc, etc)
          Hide
          aw Allen Wittenauer added a comment -

          Making this a blocker for 3.x, since it is important for 3rd party integration.

          Show
          aw Allen Wittenauer added a comment - Making this a blocker for 3.x, since it is important for 3rd party integration.

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development