Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12880

spark-assembly causes Hive class version problems

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0
    • None
    • None

    Description

      It looks like spark-assembly contains versions of Hive classes (e.g. HiveConf), and these sometimes (always?) come from older versions of Hive.
      We've seen problems where depending on classpath perturbations, NoSuchField errors may be thrown for recently added ConfVars because the HiveConf class comes from spark-assembly.

      Would making sure spark-assembly comes last in the classpath solve the problem?
      Otherwise, can we depend on something that does not package older Hive classes?

      Currently, HIVE-12179 provides a workaround (in non-Spark use case, at least; I am assuming this issue can also affect Hive-on-Spark).

      Attachments

        1. HIVE-12880.patch
          2 kB
          Sergey Shelukhin
        2. HIVE-12880.01.patch
          0.5 kB
          Sergey Shelukhin
        3. HIVE-12880.02.patch
          0.5 kB
          Sergey Shelukhin

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12179 Add option to not add spark-assembly.jar to Hive classpath

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-9608 Define SPARK_HOME if not defined automagically

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              sershe Sergey Shelukhin
              huizane Hui Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: