Hive
  1. Hive
  2. HIVE-3664

Avoid to create a symlink for hive-contrib.jar file in dist\lib folder.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.1, 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Build Infrastructure
    • Labels:

      Description

      It forces us to enumerate all the jars except this jar on Windows instead of directly referencing the “dist\lib*.jar” folder in the class path.

        Activity

        Hide
        Kanna Karanam added a comment -

        Please find the code review request at https://reviews.apache.org/r/7840/

        Show
        Kanna Karanam added a comment - Please find the code review request at https://reviews.apache.org/r/7840/
        Hide
        Ashutosh Chauhan added a comment -

        @Kanna, HIVE-2061 introduced this for backward compatibility reason. Currently, user needs to add this jar in their script. So, if we don't create a symlink user needs to update their script every time we upgrade a hive version, which is not ideal. I am not sure what problem you are facing with symlinks on Windows, but if that is not easily solvable one potential workaround would be to check in build.xml that OS is windows and then not create symlink, but instead copy hive-contrib-0.9.0 with name hive-contrib.jar. This will avoid backward incompatibility as well as need to create symlink on Windows. But, before we go down that path perhaps we should investigate more whats the problem with symlinks on Windows.

        Show
        Ashutosh Chauhan added a comment - @Kanna, HIVE-2061 introduced this for backward compatibility reason. Currently, user needs to add this jar in their script. So, if we don't create a symlink user needs to update their script every time we upgrade a hive version, which is not ideal. I am not sure what problem you are facing with symlinks on Windows, but if that is not easily solvable one potential workaround would be to check in build.xml that OS is windows and then not create symlink, but instead copy hive-contrib-0.9.0 with name hive-contrib.jar. This will avoid backward incompatibility as well as need to create symlink on Windows. But, before we go down that path perhaps we should investigate more whats the problem with symlinks on Windows.
        Hide
        Kanna Karanam added a comment -

        Thanks Ashutosh - In Windows, When we specify the class path as “HIVE_DIR\Dist\Lib*” instead of enumerating individual jars (because of 8k limitation on environment variable), JAVA is trying to load the “hive-contrib.jar” as regular jar and failing. Based on the comments in HIVE-3092, I think it is safe to remove them. If not, I can conditionally skip this on Windows.

        udf-compile:
        [echo] *** Compiling UDFs ***
        [javac] Compiling 25 source files to c:\Jenkins\workspace\windows.hive.20.unsecure\datateamtest\hcatalog\udfs\java\build
        [javac] error: error reading C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar; error in opening zip file
        [javac] 1 error
        On clicking C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar, I get an error message "Invalid or corrupt jarfile C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar"
        C:\Hadoop\hive-0.9.0\lib>ls -l hive_contrib.jar
        rw-rr- 1 Administ Administ 58 Aug 30 00:41 hive_contrib.jar

        C:\Hadoop\hive-0.9.0\lib>jar -tvf hive_contrib.jar
        java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:127)
        at java.util.zip.ZipFile.<init>(ZipFile.java:88)
        at sun.tools.jar.Main.list(Main.java:979)
        at sun.tools.jar.Main.run(Main.java:224)
        at sun.tools.jar.Main.main(Main.java:1149)

        Show
        Kanna Karanam added a comment - Thanks Ashutosh - In Windows, When we specify the class path as “HIVE_DIR\Dist\Lib*” instead of enumerating individual jars (because of 8k limitation on environment variable), JAVA is trying to load the “hive-contrib.jar” as regular jar and failing. Based on the comments in HIVE-3092 , I think it is safe to remove them. If not, I can conditionally skip this on Windows. udf-compile: [echo] *** Compiling UDFs *** [javac] Compiling 25 source files to c:\Jenkins\workspace\windows.hive.20.unsecure\datateamtest\hcatalog\udfs\java\build [javac] error: error reading C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar; error in opening zip file [javac] 1 error On clicking C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar, I get an error message "Invalid or corrupt jarfile C:\Hadoop\hive-0.9.0\lib\hive_contrib.jar" C:\Hadoop\hive-0.9.0\lib>ls -l hive_contrib.jar rw-r r - 1 Administ Administ 58 Aug 30 00:41 hive_contrib.jar C:\Hadoop\hive-0.9.0\lib>jar -tvf hive_contrib.jar java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:127) at java.util.zip.ZipFile.<init>(ZipFile.java:88) at sun.tools.jar.Main.list(Main.java:979) at sun.tools.jar.Main.run(Main.java:224) at sun.tools.jar.Main.main(Main.java:1149)
        Hide
        Ashutosh Chauhan added a comment -

        Based on discussion in HIVE-3092 it seems like no one is depending on it. So, I think we are fine removing the symlink.
        +1 will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - Based on discussion in HIVE-3092 it seems like no one is depending on it. So, I think we are fine removing the symlink. +1 will commit if tests pass.
        Hide
        Kanna Karanam added a comment -

        @Ashutosh - Can you please commit these changes if there are no UT failures? Thanks

        Show
        Kanna Karanam added a comment - @Ashutosh - Can you please commit these changes if there are no UT failures? Thanks
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Kanna!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Kanna!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1797 (See https://builds.apache.org/job/Hive-trunk-h0.21/1797/)
        HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) (Revision 1409548)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409548
        Files :

        • /hive/trunk/build.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1797 (See https://builds.apache.org/job/Hive-trunk-h0.21/1797/ ) HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) (Revision 1409548) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409548 Files : /hive/trunk/build.xml
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) (Revision 1409548)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409548
        Files :

        • /hive/trunk/build.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) (Revision 1409548) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1409548 Files : /hive/trunk/build.xml
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            Kanna Karanam
            Reporter:
            Kanna Karanam
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development