Hive
  1. Hive
  2. HIVE-2239

Use the version commons-codec from Hadoop

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0, 0.7.0, 0.8.0
    • Fix Version/s: 0.8.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hadoop upgraded from 1.3 to 1.4. Since, hive is launched from hadoop scripts, it gets 1.4 from there and then it includes 1.3 itself. Having two different versions of same jar causes unwanted grief.

      1. HIVE-2239.2.patch
        0.6 kB
        John Sichi
      2. hive-2239.patch
        1 kB
        Ashutosh Chauhan

        Issue Links

          Activity

          Hide
          Carl Steinbach added a comment -

          This patch broke the eclipse integration scripts.

          Hadoop upgraded from 1.3 to 1.4. Since, hive is launched from hadoop scripts, it gets 1.4 from there and then it includes 1.3 itself. Having two different versions of same jar causes unwanted grief.

          Hive's execution mechanism ensures that Hadoop's classpath get's preference over Hive's. Since Hive seems to work with both version 1.3 and 1.4 of commons-codec, I'm having trouble seeing how this can cause problems.

          Show
          Carl Steinbach added a comment - This patch broke the eclipse integration scripts. Hadoop upgraded from 1.3 to 1.4. Since, hive is launched from hadoop scripts, it gets 1.4 from there and then it includes 1.3 itself. Having two different versions of same jar causes unwanted grief. Hive's execution mechanism ensures that Hadoop's classpath get's preference over Hive's. Since Hive seems to work with both version 1.3 and 1.4 of commons-codec, I'm having trouble seeing how this can cause problems.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #813 (See https://builds.apache.org/job/Hive-trunk-h0.21/813/)
          HIVE-2239. Use the version of commons-codec from Hadoop
          (Ashutosh Chauhan via jvs)

          jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143565
          Files :

          • /hive/trunk/lib/commons-codec-1.3.jar
          • /hive/trunk/lib/commons-codec-1.3.LICENSE
          • /hive/trunk/build-common.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #813 (See https://builds.apache.org/job/Hive-trunk-h0.21/813/ ) HIVE-2239 . Use the version of commons-codec from Hadoop (Ashutosh Chauhan via jvs) jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143565 Files : /hive/trunk/lib/commons-codec-1.3.jar /hive/trunk/lib/commons-codec-1.3.LICENSE /hive/trunk/build-common.xml
          Hide
          John Sichi added a comment -

          Committed (+1 on my own final changes since they were trivial), thanks Ashutosh!

          Show
          John Sichi added a comment - Committed (+1 on my own final changes since they were trivial), thanks Ashutosh!
          Hide
          John Sichi added a comment -

          OK, will commit when tests pass.

          Show
          John Sichi added a comment - OK, will commit when tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Yup, your approach is cleaner then mine.
          +1

          Show
          Ashutosh Chauhan added a comment - Yup, your approach is cleaner then mine. +1
          Hide
          John Sichi added a comment -

          I was thinking we could just use the one from Hadoop directly (instead of fetching another copy from ivy), as in the attached HIVE-2239.2.patch. I verified build as well as TestContribCliDriver with qfile=fileformat_base64.q.

          Show
          John Sichi added a comment - I was thinking we could just use the one from Hadoop directly (instead of fetching another copy from ivy), as in the attached HIVE-2239 .2.patch. I verified build as well as TestContribCliDriver with qfile=fileformat_base64.q.
          Hide
          Ashutosh Chauhan added a comment -

          Ok, at runtime we don't want to include any jar file which hadoop anyway will include in its classpath. For that we simply need to delete the existing jar file from $

          {hive-root}

          /lib and we will be good. However, doing so will make hive fail to compile. So, this patch retrieves the jar and put it in the build/ivy/lib/default so that it could be used at compile time (and also while running tests). This I think is the cleanest approach, since hive doesn't force any version and whichever hadoop is running it will use its version of jar file. I upped the version in ivy/libraries.properties since we need to up it one day in any case, but strictly that is not required. After applying this patch, please necessarily delete lib/common-codecs-1.3.jar

          Show
          Ashutosh Chauhan added a comment - Ok, at runtime we don't want to include any jar file which hadoop anyway will include in its classpath. For that we simply need to delete the existing jar file from $ {hive-root} /lib and we will be good. However, doing so will make hive fail to compile. So, this patch retrieves the jar and put it in the build/ivy/lib/default so that it could be used at compile time (and also while running tests). This I think is the cleanest approach, since hive doesn't force any version and whichever hadoop is running it will use its version of jar file. I upped the version in ivy/libraries.properties since we need to up it one day in any case, but strictly that is not required. After applying this patch, please necessarily delete lib/common-codecs-1.3.jar
          Hide
          John Sichi added a comment -

          In the 0.20.1 Hadoop version we currently pull in for default Hive builds, the version is currently 1.3. To avoid conflicts there, would it be possible to instead do a patch which just uses the version from Hadoop via commons-codec-*.jar (and then also delete the one from Hive as part of this commit)?

          Show
          John Sichi added a comment - In the 0.20.1 Hadoop version we currently pull in for default Hive builds, the version is currently 1.3. To avoid conflicts there, would it be possible to instead do a patch which just uses the version from Hadoop via commons-codec-*.jar (and then also delete the one from Hive as part of this commit)?
          Hide
          Ashutosh Chauhan added a comment -

          Any word on this one?

          Show
          Ashutosh Chauhan added a comment - Any word on this one?
          Hide
          Ashutosh Chauhan added a comment -

          I tested with commons-codec-1.4 and all the tests in contrib passed.

          Show
          Ashutosh Chauhan added a comment - I tested with commons-codec-1.4 and all the tests in contrib passed.
          Hide
          Ashutosh Chauhan added a comment -

          It was introduced as a dependency in https://issues.apache.org/jira/browse/HIVE-639 Seems like its not used in core hive anywhere only in contrib. Contrib dependencies in main build perhaps should not be there in first place, but now before removing we need to deprecate them.
          Its there in 0.21 https://issues.apache.org/jira/browse/HADOOP-6579 and has been backported to 20.203 as well http://hadoop.apache.org/common/docs/r0.20.203.0/changes.html so its a good time to upgrade this jar.

          Show
          Ashutosh Chauhan added a comment - It was introduced as a dependency in https://issues.apache.org/jira/browse/HIVE-639 Seems like its not used in core hive anywhere only in contrib. Contrib dependencies in main build perhaps should not be there in first place, but now before removing we need to deprecate them. Its there in 0.21 https://issues.apache.org/jira/browse/HADOOP-6579 and has been backported to 20.203 as well http://hadoop.apache.org/common/docs/r0.20.203.0/changes.html so its a good time to upgrade this jar.

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development