Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21433

Spark SQL should support higher version of Hive metastore

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None
    • Environment:

      software platform: Spark 2.2.0, Hive metastore 1.2.1, Hadoop 2.6.5
      operating system: CentOS 6.9

    • Flags:
      Important

      Description

      Now. Spark SQL supports Hive metastore versions ranging from 0.12.0 to 1.2.1 (inclusive).
      When I start the Thrift JDBC/ODBC server and test with JDBC codes, I will discover that some running Spark Jobs is very slowly. I want to stop running code to stop this running Spark Job. But the running code stopped and the running Spark Job does not stop. The system resource can not be released.
      Is running spark jobs stopped when a query is cancelled? I think yes.
      I checked same Hive issues and Hive will resolve this issue.

      This is My test code:

      object App_2017071100 {
      
        private[this] val DRIER = "org.apache.hive.jdbc.HiveDriver"
        private[this] val URL = "jdbc:hive2://192.168.50.3:10000/default"
        private[this] val USERNAME = "hive"
        private[this] val PASSWORD = "hive"
      
        def main(args: Array[String]): Unit = {
          var connect: Connection = null
          var statement: PreparedStatement = null
          var result: ResultSet = null
          try {
            Class.forName(DRIER)
            connect = DriverManager.getConnection(URL, USERNAME, PASSWORD)
            statement = connect.prepareStatement("select count(1) from ip")
            //  statement.setQueryTimeout(30)
            result = statement.executeQuery
            while (result.next) {
              println(result.getString(1))
            }
          } catch {
            case e: LinkageError => e.printStackTrace
            case e: ExceptionInInitializerError => e.printStackTrace
            case e: ClassNotFoundException => e.printStackTrace
            case e: SQLException => e.printStackTrace
            case e: SQLTimeoutException => e.printStackTrace
          } finally {
            if (result != null) result.close
            if (statement != null) statement.close
            if (connect != null) connect.close
          }
        }
      
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Gu Chao Gu Chao
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: