Hive
  1. Hive
  2. HIVE-2998 Making Hive run on Windows Server and Windows Azure environment
  3. HIVE-3172

Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Tests, Windows
    • Labels:
    • Environment:

      Windows

      Description

      The maximum length of the DOS command string is 8191 characters (in Windows latest versions http://support.microsoft.com/kb/830473). Following entries in the “build-common.xml” are adding lot of duplicate JAR entries to the “test.classpath” and it exceeds the max character limit on windows very easily.

      <!-- Include build/dist/lib on the classpath before Ivy and exclude hive jars from Ivy to make sure we get the local changes when we test Hive -->
      <fileset dir="$

      {build.dir.hive}

      /dist/lib" includes=".jar" erroronmissingdir="false" excludes="/hive_contrib.jar,*/hive-contrib.jar,*/lib.jar"/>
      <fileset dir="$

      {hive.root}/build/ivy/lib/test" includes=".jar" erroronmissingdir="false" excludes="/hive_.jar,*/hive-.jar"/>
      <fileset dir="${hive.root}

      /build/ivy/lib/default" includes=".jar" erroronmissingdir="false" excludes="/hive_.jar,*/hive-.jar" />
      <fileset dir="$

      {hive.root}

      /testlibs" includes="*.jar"/>

      Proposed solution (workaround)–
      1)Include all JARs from dist\lib excluding "*/hive_contrib.jar,*/hive-contrib.jar,*/lib.jar"
      2)Select the specific jars (missing jars) from test/other folders, (that includes Hadoop-*.jar files)

      Thanks

      1. HIVE-3172.3.patch.txt
        13 kB
        Kanna Karanam
      2. HIVE-3172.2.patch.txt
        14 kB
        Kanna Karanam
      3. HIVE-3172.1.patch.txt
        9 kB
        Kanna Karanam

        Issue Links

          Activity

          Hide
          Carl Steinbach added a comment -

          Select the specific jars (missing jars) from test/other folders, (that includes Hadoop-*.jar files)

          How are you planning to do this?

          Also, which command is failing because of the DOS string length limit?

          Show
          Carl Steinbach added a comment - Select the specific jars (missing jars) from test/other folders, (that includes Hadoop-*.jar files) How are you planning to do this? Also, which command is failing because of the DOS string length limit?
          Hide
          Kanna Karanam added a comment -

          @Carl - All MiniMR cluster related test cases are failing when hadoop is attempting to run the MR jobs.
          I am debating on following two approaches-.

          1) Create a custom ANT task that takes the list of folders/filesets as an input and enumerates files from each folder and adds to list/classpath if doesn’t exist already. (Based on unique names). I have very limited knowledge on ANT so please let me know if you see any error.
          2) Hacky solution is replace the fileset with path element in the test class path.

          Please let me know if you see any other possible solutions.

          Thanks

          Show
          Kanna Karanam added a comment - @Carl - All MiniMR cluster related test cases are failing when hadoop is attempting to run the MR jobs. I am debating on following two approaches-. 1) Create a custom ANT task that takes the list of folders/filesets as an input and enumerates files from each folder and adds to list/classpath if doesn’t exist already. (Based on unique names). I have very limited knowledge on ANT so please let me know if you see any error. 2) Hacky solution is replace the fileset with path element in the test class path. Please let me know if you see any other possible solutions. Thanks
          Hide
          Kanna Karanam added a comment -

          @Carl – I was thinking of another easy and cleaner approach to solve this. If we can drop all jars to a single folder and refer that folder in the class path instead of multiple folders. Please let me know your opinion on this approach. Thanks

          Show
          Kanna Karanam added a comment - @Carl – I was thinking of another easy and cleaner approach to solve this. If we can drop all jars to a single folder and refer that folder in the class path instead of multiple folders. Please let me know your opinion on this approach. Thanks
          Hide
          Kanna Karanam added a comment -

          Attached the patch.

          Show
          Kanna Karanam added a comment - Attached the patch.
          Hide
          Kanna Karanam added a comment -

          Extended the ANT path class to remove the duplicate JAR entries from the original class path.

          Show
          Kanna Karanam added a comment - Extended the ANT path class to remove the duplicate JAR entries from the original class path.
          Hide
          Kanna Karanam added a comment -

          Updated the patch with missing files.

          Show
          Kanna Karanam added a comment - Updated the patch with missing files.
          Hide
          Kanna Karanam added a comment -

          code review request is available at https://reviews.apache.org/r/5541/

          Show
          Kanna Karanam added a comment - code review request is available at https://reviews.apache.org/r/5541/
          Hide
          Carl Steinbach added a comment -

          > All MiniMR cluster related test cases are failing when hadoop is attempting to run the MR jobs.

          I'm kind of surprised that this isn't breaking all of the tests. Why does it only affect the Minimr tests? Specifically which parameter is too long? Can you please provide a reference to a class and method? Thanks.

          Show
          Carl Steinbach added a comment - > All MiniMR cluster related test cases are failing when hadoop is attempting to run the MR jobs. I'm kind of surprised that this isn't breaking all of the tests. Why does it only affect the Minimr tests? Specifically which parameter is too long? Can you please provide a reference to a class and method? Thanks.
          Hide
          Kanna Karanam added a comment -

          @Carl - You are right. It has impact on all unit tests (Not just Minimr). Length of the %CLASSPATH% is almost ~14000 characters length with duplicate entries. It came down to ~ 6000 characters after removing the duplicate entries.

          HIVE-3126 changes are required to run unit tests on Windows and HIVE-3126.1.patch.txt had a work around to that masked all other unit test failures. I removed that workaround in HIVE-3126.2.patch.txt

          Show
          Kanna Karanam added a comment - @Carl - You are right. It has impact on all unit tests (Not just Minimr). Length of the %CLASSPATH% is almost ~14000 characters length with duplicate entries. It came down to ~ 6000 characters after removing the duplicate entries. HIVE-3126 changes are required to run unit tests on Windows and HIVE-3126 .1.patch.txt had a work around to that masked all other unit test failures. I removed that workaround in HIVE-3126 .2.patch.txt
          Hide
          Ashutosh Chauhan added a comment -

          Hi Kanna,
          I left some review comments on Review Board. Thanks!

          Show
          Ashutosh Chauhan added a comment - Hi Kanna, I left some review comments on Review Board. Thanks!
          Hide
          Kanna Karanam added a comment -

          Removed the unused imports from the previous patch.

          Show
          Kanna Karanam added a comment - Removed the unused imports from the previous patch.
          Hide
          Ashutosh Chauhan added a comment -

          +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - +1 will commit if tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Kanna!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Kanna!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1522 (See https://builds.apache.org/job/Hive-trunk-h0.21/1522/)
          HIVE-3172 : Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows (Kanna Karanam via Ashutosh Chauhan) (Revision 1356035)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356035
          Files :

          • /hive/trunk/ant/src/org/apache/hadoop/hive/ant/DistinctElementsClassPath.java
          • /hive/trunk/build-common.xml
          • /hive/trunk/jdbc/build.xml
          • /hive/trunk/odbc/build.xml
          • /hive/trunk/shims/build.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1522 (See https://builds.apache.org/job/Hive-trunk-h0.21/1522/ ) HIVE-3172 : Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows (Kanna Karanam via Ashutosh Chauhan) (Revision 1356035) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356035 Files : /hive/trunk/ant/src/org/apache/hadoop/hive/ant/DistinctElementsClassPath.java /hive/trunk/build-common.xml /hive/trunk/jdbc/build.xml /hive/trunk/odbc/build.xml /hive/trunk/shims/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3172 : Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows (Kanna Karanam via Ashutosh Chauhan) (Revision 1356035)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356035
          Files :

          • /hive/trunk/ant/src/org/apache/hadoop/hive/ant/DistinctElementsClassPath.java
          • /hive/trunk/build-common.xml
          • /hive/trunk/jdbc/build.xml
          • /hive/trunk/odbc/build.xml
          • /hive/trunk/shims/build.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3172 : Remove the duplicate JAR entries from the (“test.classpath”) to avoid command line exceeding char limit on windows (Kanna Karanam via Ashutosh Chauhan) (Revision 1356035) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356035 Files : /hive/trunk/ant/src/org/apache/hadoop/hive/ant/DistinctElementsClassPath.java /hive/trunk/build-common.xml /hive/trunk/jdbc/build.xml /hive/trunk/odbc/build.xml /hive/trunk/shims/build.xml
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              Kanna Karanam
              Reporter:
              Kanna Karanam
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development