Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12880

spark-assembly causes Hive class version problems

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • None
    • None

    Description

      It looks like spark-assembly contains versions of Hive classes (e.g. HiveConf), and these sometimes (always?) come from older versions of Hive.
      We've seen problems where depending on classpath perturbations, NoSuchField errors may be thrown for recently added ConfVars because the HiveConf class comes from spark-assembly.

      Would making sure spark-assembly comes last in the classpath solve the problem?
      Otherwise, can we depend on something that does not package older Hive classes?

      Currently, HIVE-12179 provides a workaround (in non-Spark use case, at least; I am assuming this issue can also affect Hive-on-Spark).

      Attachments

        1. HIVE-12880.patch
          2 kB
          Sergey Shelukhin
        2. HIVE-12880.02.patch
          0.5 kB
          Sergey Shelukhin
        3. HIVE-12880.01.patch
          0.5 kB
          Sergey Shelukhin

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sershe Sergey Shelukhin Assign to me
            huizane Hui Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment