Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10793

Make spark's use/subclassing of hive more maintainable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      The latest spark/hive integration round has closed the gap with Hive versions, but the integration is still pretty complex

      1. SparkSQL has deep hooks into the parser
      2. hivethriftserver uses "aggressive reflection" to inject spark classes into the Hive base classes.
      3. there's a separate org.sparkproject.hive JAR to isolate Kryo versions while avoiding the hive uberjar with all its dependencies getting into the spark uberjar.

      We can improve this with some assistance from the other projects, even though no guarantees of stability of things like the parser and thrift server APIs are likely in the near future

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: