Bigtop
  1. Bigtop
  2. BIGTOP-1004

Hadoop packages do not include version independent symlinks

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      The Hadoop packages should install version independent symlinks.

      When working on BIGTOP-993 I couldn't replace Hadoop dependencies in the component lib/ with symlinks, unlike the case for Zookeeper and HBase, because version independent symlinks aren't available for Hadoop jars.

      1. BIGTOP-1004.patch
        2 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          There is actually this in install_hadoop.sh meant to create version independent symlinks:

          # Remove all source and create version-less symlinks to offer integration point with other projects
          for DIR in ${HADOOP_DIR} ${HDFS_DIR} ${YARN_DIR} ${MAPREDUCE_DIR} ${HTTPFS_DIR} ; do
            (cd $DIR &&
             rm -fv *-sources.jar
             rm -fv lib/hadoop-*.jar
             for j in hadoop-*.jar; do
               if [[ $j =~ hadoop-(.*)-${HADOOP_VERSION}.jar ]]; then
                 name=${BASH_REMATCH[1]}
                 ln -s $j hadoop-$name.jar
               fi
             done)
          done
          

          However, the symlinks are not present as expected after installation.

          The problem is HADOOP_VERSION is hardcoded in install_hadoop.sh to "0.21.3".

          + for j in 'hadoop-*.jar'
          + [[ hadoop-annotations-2.0.5-alpha.jar =~ hadoop-(.*)-0.23.1.jar ]]
          ...
          

          Attached a patch that sets up HADOOP_VERSION in the environment. Confirmed the presence of expected symlinks after package install.

          Show
          Andrew Purtell added a comment - There is actually this in install_hadoop.sh meant to create version independent symlinks: # Remove all source and create version-less symlinks to offer integration point with other projects for DIR in ${HADOOP_DIR} ${HDFS_DIR} ${YARN_DIR} ${MAPREDUCE_DIR} ${HTTPFS_DIR} ; do (cd $DIR && rm -fv *-sources.jar rm -fv lib/hadoop-*.jar for j in hadoop-*.jar; do if [[ $j =~ hadoop-(.*)-${HADOOP_VERSION}.jar ]]; then name=${BASH_REMATCH[1]} ln -s $j hadoop-$name.jar fi done) done However, the symlinks are not present as expected after installation. The problem is HADOOP_VERSION is hardcoded in install_hadoop.sh to "0.21.3". + for j in 'hadoop-*.jar' + [[ hadoop-annotations-2.0.5-alpha.jar =~ hadoop-(.*)-0.23.1.jar ]] ... Attached a patch that sets up HADOOP_VERSION in the environment. Confirmed the presence of expected symlinks after package install.
          Hide
          Sean Mackrory added a comment -

          +1 (non-committer)

          Show
          Sean Mackrory added a comment - +1 (non-committer)
          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!

            People

            • Assignee:
              Andrew Purtell
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development