Hadoop Common
  1. Hadoop Common
  2. HADOOP-7615

Binary layout does not put share/hadoop/contrib/*.jar into the class path

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0, 0.23.0
    • Fix Version/s: 0.20.205.0
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Java, Linux

      Description

      For contrib projects, contrib jar files are not included in HADOOP_CLASSPATH in the binary layout. Several projects jar files should be copied to $HADOOP_PREFIX/share/hadoop/lib for binary deployment. The interesting jar files to include in $HADOOP_PREFIX/share/hadoop/lib are: capacity-scheduler, thriftfs, fairscheduler.

        Activity

        Hide
        Joep Rottinghuis added a comment -

        Are you proposing to leave them in their current location and add them to the classpath through hadoop-env.sh, or to move them into common/lib for 0.20 and hadoop-mapreduce/lib for 0.22, 0.23, and beyond?
        Should we anticipate any side-effects ( for example to have both schedulers on the CP)?

        Show
        Joep Rottinghuis added a comment - Are you proposing to leave them in their current location and add them to the classpath through hadoop-env.sh, or to move them into common/lib for 0.20 and hadoop-mapreduce/lib for 0.22, 0.23, and beyond? Should we anticipate any side-effects ( for example to have both schedulers on the CP)?
        Hide
        Eric Yang added a comment -

        Owen recommended to put the jar files into $HADOOP_PREFIX/share/hadoop/$project/lib to reduce custom script to construct class path for contrib projects. For capacity-scheduler and fair scheduler, the package path are clean and independent from each other, there shouldn't be any issue to have both on the class path.

        Show
        Eric Yang added a comment - Owen recommended to put the jar files into $HADOOP_PREFIX/share/hadoop/$project/lib to reduce custom script to construct class path for contrib projects. For capacity-scheduler and fair scheduler, the package path are clean and independent from each other, there shouldn't be any issue to have both on the class path.
        Hide
        Eric Yang added a comment -

        Customize thrift, fair scheduler, and capacity scheduler to put their jar files in $PREFIX/share/hadoop/lib for binary layout, or $PREFIX/lib for tar layout.

        Show
        Eric Yang added a comment - Customize thrift, fair scheduler, and capacity scheduler to put their jar files in $PREFIX/share/hadoop/lib for binary layout, or $PREFIX/lib for tar layout.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494273/HADOOP-7615.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/171//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494273/HADOOP-7615.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/171//console This message is automatically generated.
        Hide
        Matt Foley added a comment -

        +1 lgtm for 20-sec and 205. Re trunk, Eric says, "This is currently not possible to generate for trunk due to yarn packaging and mapreduce structure layout is not finalized/working."

        Show
        Matt Foley added a comment - +1 lgtm for 20-sec and 205. Re trunk, Eric says, "This is currently not possible to generate for trunk due to yarn packaging and mapreduce structure layout is not finalized/working."
        Hide
        Devaraj Das added a comment -

        Committed this to the branch-0.20-security and branch-0.20-security-205 branches. Keeping the issue open to track the commit on trunk.

        Show
        Devaraj Das added a comment - Committed this to the branch-0.20-security and branch-0.20-security-205 branches. Keeping the issue open to track the commit on trunk.
        Hide
        Mit Desai added a comment -

        I see the target version here is pretty old and there is no activity on the Jira since a long time. Does this still need to be in 0.23? We can re-target this if so or else close the issue.

        Show
        Mit Desai added a comment - I see the target version here is pretty old and there is no activity on the Jira since a long time. Does this still need to be in 0.23? We can re-target this if so or else close the issue.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494273/HADOOP-7615.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3782//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494273/HADOOP-7615.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3782//console This message is automatically generated.
        Hide
        Eric Yang added a comment -

        On 0.23 branch, there is no contrib beside fuse in hadoop hdfs project. Structure has mainly changed due to project layout has been refactored by maven project build of Hadoop. Fuse is the only contrib project in hdfs, and it doesn't need to be on the default class path of Hadoop. Hence, close this as resolved.

        Show
        Eric Yang added a comment - On 0.23 branch, there is no contrib beside fuse in hadoop hdfs project. Structure has mainly changed due to project layout has been refactored by maven project build of Hadoop. Fuse is the only contrib project in hdfs, and it doesn't need to be on the default class path of Hadoop. Hence, close this as resolved.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development