Hive
  1. Hive
  2. HIVE-5733

Publish hive-exec artifact without all the dependencies

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently the artifact hive-exec that is available in maven is shading all the dependencies (= the jar contains all Hive's dependencies). As other projects that are depending on Hive might be use slightly different version of the dependencies, it can easily happens that Hive's shaded version will be used instead which leads to very time consuming debugging of what is happening (for example SQOOP-1198).

      Would it be feasible publish hive-exec jar that will be build without shading any dependency? For example avro-tools is having classifier "nodeps" that represents artifact without any dependencies.

      1. HIVE-5733.1.patch
        0.8 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Patrick Wendell added a comment -

          Hey just wanted to add a +1 and say that the current approach makes depending on Hive difficult or impossible for certain Hadoop versions due to conflicts with the protobuf library.

          Show
          Patrick Wendell added a comment - Hey just wanted to add a +1 and say that the current approach makes depending on Hive difficult or impossible for certain Hadoop versions due to conflicts with the protobuf library.
          Hide
          Amareshwari Sriramadasu added a comment -

          +1 This is much required.
          I agree it has become difficult to depend on hive exec jar, because of ql module shading all the dependencies.

          I will try to put a patch.

          Show
          Amareshwari Sriramadasu added a comment - +1 This is much required. I agree it has become difficult to depend on hive exec jar, because of ql module shading all the dependencies. I will try to put a patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          As per the documentation of maven-shade-plugin the plugin will replace the project's main artifact with the shaded artifact. Giving a different classifier like nodep also was not helpful. As per the doc, the shaded artifact can be given a different name. Doc - http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-artifact.html

          Attaching the patch which generates hive-exec-<version>.jar with no dependencies and hive-exec-<version>.-withdep.jar as the shaded jar.

          Show
          Amareshwari Sriramadasu added a comment - As per the documentation of maven-shade-plugin the plugin will replace the project's main artifact with the shaded artifact. Giving a different classifier like nodep also was not helpful. As per the doc, the shaded artifact can be given a different name. Doc - http://maven.apache.org/plugins/maven-shade-plugin/examples/attached-artifact.html Attaching the patch which generates hive-exec-<version>.jar with no dependencies and hive-exec-<version>.-withdep.jar as the shaded jar.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12644972/HIVE-5733.1.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/200/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/200/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12644972

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12644972/HIVE-5733.1.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/200/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/200/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12644972
          Hide
          Amareshwari Sriramadasu added a comment -

          There are problems with uploaded patch, canceling

          Show
          Amareshwari Sriramadasu added a comment - There are problems with uploaded patch, canceling
          Hide
          Steve Loughran added a comment -

          Amareshwari Sriramadasu : what problems did you see with this patch? And have you had a go at doing another iteration to fix them?

          Show
          Steve Loughran added a comment - Amareshwari Sriramadasu : what problems did you see with this patch? And have you had a go at doing another iteration to fix them?
          Hide
          Amareshwari Sriramadasu added a comment -

          what problems did you see with this patch?

          I think the problem was - "removing shaded dependencies from hive-exec caused failures in other modules, as other modules are expecting all dependencies come with hive-exec". Other solution we did on our end is to make protobuf version provided and not bundle with hive-exec jar.

          Show
          Amareshwari Sriramadasu added a comment - what problems did you see with this patch? I think the problem was - "removing shaded dependencies from hive-exec caused failures in other modules, as other modules are expecting all dependencies come with hive-exec". Other solution we did on our end is to make protobuf version provided and not bundle with hive-exec jar.
          Hide
          Steve Loughran added a comment -

          I see. Protobuf is probably the JAR most brittle across versions; Guava being the other. If you are doing provided, there's a risk things won't link. At least it will fail at classload time —it's not internal behaviour that's changed silently.

          Show
          Steve Loughran added a comment - I see. Protobuf is probably the JAR most brittle across versions; Guava being the other. If you are doing provided, there's a risk things won't link. At least it will fail at classload time —it's not internal behaviour that's changed silently.
          Hide
          Steve Loughran added a comment -

          I'm going to close this as there is an artifact without dependencies being built, you refer to it with the classifier <classifier>core</classifier>. You do then get to choose the exact set of dependencies, at the expense of working out which are compatible.

          Closing as fixed in 1.2.0; don't know when the fix went in

          Show
          Steve Loughran added a comment - I'm going to close this as there is an artifact without dependencies being built, you refer to it with the classifier <classifier>core</classifier> . You do then get to choose the exact set of dependencies, at the expense of working out which are compatible. Closing as fixed in 1.2.0; don't know when the fix went in
          Hide
          Thejas M Nair added a comment -

          Closing jiras fixed in 1.2.0 release .
          Please open new jira if you find any related issue.

          Show
          Thejas M Nair added a comment - Closing jiras fixed in 1.2.0 release . Please open new jira if you find any related issue.

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Jarek Jarcec Cecho
            • Votes:
              8 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development