Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7950

HiveThriftServer2.startWithContext() doesn't set "spark.sql.hive.version"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.2.1, 1.2.2, 1.3.0, 1.3.1, 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: SQL
    • Labels:
      None
    • Environment:

      Simba Spark SQL ODBC driver 1.0.8.1006

    • Target Version/s:

      Description

      While testing the newly released Simba Spark SQL ODBC driver 1.0.8.1006 against 1.4.0-SNAPSHOT, we found that if HiveThriftServer2 is started with HiveThriftServer2.startWithContext(), then simple queries like

      SELECT * FROM src
      

      fail with the following error message (need to turn on ODBC trace log):

      DIAG [S0002] [Simba][SQLEngine] (31740) Table or view not found: SPARK..src
      

      However, JDBC client like Beeline is fine. Also, if the server is started via sbin/start-thriftserver.sh, both ODBC and JDBC work fine.

      The reason for this failure is that, HiveThriftServer2.startWithContext() doesn't properly set the "spark.sql.hive.version" property. It seems that Simba ODBC driver 1.0.8.1006 behaves differently when this property is missing. What I observed is that, in this case, the ODBC driver issues a GetColumns command, which isn't overriden in Spark HiveThriftServer2, and this falls back to original Hive code path, which results in unexpected behavior.

        Attachments

          Activity

            People

            • Assignee:
              lian cheng Cheng Lian
              Reporter:
              lian cheng Cheng Lian
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: