Hadoop Common
  1. Hadoop Common
  2. HADOOP-8125

make hadoop-client set of curated jars available in a distribution tarball

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      One thing that the original patch for HADOOP-8082 didn't address is the need for those curated jars to be visible in the final tarball.

      1. HADOOP-8125-4.patch.txt
        1 kB
        Roman Shaposhnik
      2. HADOOP-8125-3.patch.txt
        3 kB
        Roman Shaposhnik
      3. HADOOP-8125-2.patch.txt
        4 kB
        Roman Shaposhnik
      4. HADOOP-8125.patch.txt
        4 kB
        Roman Shaposhnik

        Activity

        Hide
        Roman Shaposhnik added a comment -

        Attaching a patch that creates share/hadoop/client in the final tarball and populate it with symlinks to the actual jar files.

        It is now safe for downstream components to simply add share/hadoop/client and share/hadoop/client/lib to their classpath without getting extra dependencies that can be dangerous in a client type environment (like getting jettty/tomcat stuff)

        Show
        Roman Shaposhnik added a comment - Attaching a patch that creates share/hadoop/client in the final tarball and populate it with symlinks to the actual jar files. It is now safe for downstream components to simply add share/hadoop/client and share/hadoop/client/lib to their classpath without getting extra dependencies that can be dangerous in a client type environment (like getting jettty/tomcat stuff)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516636/HADOOP-8125.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/648//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516636/HADOOP-8125.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/648//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        Roman, the assembly file should be in the hadoop-assembly module. Then you have to configure the assembly-plugin in hadoop-client to include hadoop-assembly as dependency (see how hadoop-project-dist/pom.xml does it). I've tested the patch on OSX and the share/hadoop/client/lib/ dir of the dist is empty.

        Show
        Alejandro Abdelnur added a comment - Roman, the assembly file should be in the hadoop-assembly module. Then you have to configure the assembly-plugin in hadoop-client to include hadoop-assembly as dependency (see how hadoop-project-dist/pom.xml does it). I've tested the patch on OSX and the share/hadoop/client/lib/ dir of the dist is empty.
        Hide
        Roman Shaposhnik added a comment -

        Attaching a second version of the patch that takes care of all the raised concerns. Tested with:

        $ mvn clean package -DskipTests -Pdist
        $ ls -l hadoop-dist/target/hadoop*/share/hadoop/client/
        

        on Ubuntu Linux

        Show
        Roman Shaposhnik added a comment - Attaching a second version of the patch that takes care of all the raised concerns. Tested with: $ mvn clean package -DskipTests -Pdist $ ls -l hadoop-dist/target/hadoop*/share/hadoop/client/ on Ubuntu Linux
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516720/HADOOP-8125-2.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/651//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516720/HADOOP-8125-2.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/651//console This message is automatically generated.
        Hide
        Roman Shaposhnik added a comment -

        Attaching a minimalistic version of the patch that has an assembly creation part but doesn't push the bits into the final tarball.

        For the Bigtop purposes either one of the patches would suffice. Feel free to pick one that works better for the Hadoop community.

        Show
        Roman Shaposhnik added a comment - Attaching a minimalistic version of the patch that has an assembly creation part but doesn't push the bits into the final tarball. For the Bigtop purposes either one of the patches would suffice. Feel free to pick one that works better for the Hadoop community.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516745/HADOOP-8125-3.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The patch appears to cause tar ant target to fail.

        -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed the unit tests build

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516745/HADOOP-8125-3.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//console This message is automatically generated.
        Hide
        Roman Shaposhnik added a comment -

        A colleague of mine suggested even simpler solution then all of the above. Attaching.

        Show
        Roman Shaposhnik added a comment - A colleague of mine suggested even simpler solution then all of the above. Attaching.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516918/HADOOP-8125-4.patch.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/666//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516918/HADOOP-8125-4.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/666//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        +1

        Show
        Alejandro Abdelnur added a comment - +1
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Roman. Committed to trunk and branch-0.23.

        I've committed an amendment to trunk as the relativePath fo the parent was incorrect and I have not noticed as I had a copy of the parent in my M2 cache.

        Show
        Alejandro Abdelnur added a comment - Thanks Roman. Committed to trunk and branch-0.23. I've committed an amendment to trunk as the relativePath fo the parent was incorrect and I have not noticed as I had a copy of the parent in my M2 cache.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/)
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1837 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/ ) HADOOP-8125 . make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270 Files : /hadoop/common/trunk/hadoop-client/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #631 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/631/)
        Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #631 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/631/ ) Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281 Files : /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/641/)
        Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #641 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/641/ ) Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281 Files : /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/)
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/ ) HADOOP-8125 . make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270 Files : /hadoop/common/trunk/hadoop-client/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/)
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/ ) HADOOP-8125 . make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270 Files : /hadoop/common/trunk/hadoop-client/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #642 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/642/)
        Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #642 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/642/ ) Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281 Files : /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #189 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/189/)
        Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #189 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/189/ ) Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281 Files : /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/)
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #976 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/ ) HADOOP-8125 . make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270 Files : /hadoop/common/trunk/hadoop-client/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/217/)
        Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/217/ ) Merge -r 1297269:1297270 from trunk to branch. FIXES: HADOOP-8125 (Revision 1297281) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297281 Files : /hadoop/common/branches/branch-0.23/hadoop-client/pom.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/)
        HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270
        Files :

        • /hadoop/common/trunk/hadoop-client/pom.xml
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1011 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/ ) HADOOP-8125 . make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu) (Revision 1297270) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297270 Files : /hadoop/common/trunk/hadoop-client/pom.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development