[SPARK-21433] Spark SQL should support higher version of Hive metastore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.0
• Component/s: SQL

Environment

software platform: Spark 2.2.0, Hive metastore 1.2.1, Hadoop 2.6.5
operating system: CentOS 6.9

• Flags: Important
• Assignee: Unassigned
• Reporter: Gu Chao
• Created: 17/Jul/17 03:29
• Updated: 19/Jul/17 08:57
• Resolved: 19/Jul/17 08:57

Description

Now. Spark SQL supports Hive metastore versions ranging from 0.12.0 to 1.2.1 (inclusive).
When I start the Thrift JDBC/ODBC server and test with JDBC codes, I will discover that some running Spark Jobs is very slowly. I want to stop running code to stop this running Spark Job. But the running code stopped and the running Spark Job does not stop. The system resource can not be released.
Is running spark jobs stopped when a query is cancelled? I think yes.
I checked same Hive issues and Hive will resolve this issue.

This is My test code:

object App_2017071100 {

  private[this] val DRIER = "org.apache.hive.jdbc.HiveDriver"
  private[this] val URL = "jdbc:hive2://192.168.50.3:10000/default"
  private[this] val USERNAME = "hive"
  private[this] val PASSWORD = "hive"

  def main(args: Array[String]): Unit = {
    var connect: Connection = null
    var statement: PreparedStatement = null
    var result: ResultSet = null
    try {
      Class.forName(DRIER)
      connect = DriverManager.getConnection(URL, USERNAME, PASSWORD)
      statement = connect.prepareStatement("select count(1) from ip")
      //  statement.setQueryTimeout(30)
      result = statement.executeQuery
      while (result.next) {
        println(result.getString(1))
      }
    } catch {
      case e: LinkageError => e.printStackTrace
      case e: ExceptionInInitializerError => e.printStackTrace
      case e: ClassNotFoundException => e.printStackTrace
      case e: SQLException => e.printStackTrace
      case e: SQLTimeoutException => e.printStackTrace
    } finally {
      if (result != null) result.close
      if (statement != null) statement.close
      if (connect != null) connect.close
    }
  }

}

Issue Links

• is related to: HIVE-16422 Should kill running Spark Jobs when a query is cancelled.
• is related to: HIVE-13760 Add a HIVE_QUERY_TIMEOUT configuration to kill a query if a query is running for more than the configured timeout value.
• is related to: HIVE-4924 JDBC: Support query timeout for jdbc