[SPARK-13983] HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version (both multi-session and single session)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0, 1.6.1
• Fix Version/s: 2.3.0
• Component/s: SQL

Environment

ubuntu, spark 1.6.0 standalone, spark 1.6.1 standalone
(tried spark branch-1.6 snapshot as well)
compiled with scala 2.10.5 and hadoop 2.6
(-Phadoop-2.6 -Psparkr -Phive -Phive-thriftserver)

• Assignee: Yuming Wang
• Reporter: Teng Qiu
• Created: 17/Mar/16 20:17
• Updated: 01/Feb/18 18:37
• Resolved: 01/Feb/18 18:37

Description

HiveThriftServer2 should be able to get "--hiveconf" or ''--hivevar" variables from JDBC client, either from command line parameter of beeline, such as
beeline --hiveconf spark.sql.shuffle.partitions=3 --hivevar db_name=default
or from JDBC connection string, like
jdbc:hive2://localhost:10000?spark.sql.shuffle.partitions=3#db_name=default

this worked in spark version 1.5.x, but after upgraded to 1.6, it doesn't work.

to reproduce this issue, try to connect to HiveThriftServer2 with beeline:

bin/beeline -u jdbc:hive2://localhost:10000 \
            --hiveconf spark.sql.shuffle.partitions=3 \
            --hivevar db_name=default

or

bin/beeline -u jdbc:hive2://localhost:10000?spark.sql.shuffle.partitions=3#db_name=default

will get following results:

0: jdbc:hive2://localhost:10000> set spark.sql.shuffle.partitions;
+-------------------------------+--------+--+
|              key              | value  |
+-------------------------------+--------+--+
| spark.sql.shuffle.partitions  | 200    |
+-------------------------------+--------+--+
1 row selected (0.192 seconds)
0: jdbc:hive2://localhost:10000> use ${db_name};
Error: org.apache.spark.sql.AnalysisException: cannot recognize input near '$' '{' 'db_name' in switch database statement; line 1 pos 4 (state=,code=0)

-

but this bug does not affect current versions of spark-sql CLI, following commands works:

bin/spark-sql --master local[2] \
              --hiveconf spark.sql.shuffle.partitions=3 \
              --hivevar db_name=default

spark-sql> set spark.sql.shuffle.partitions
spark.sql.shuffle.partitions   3
Time taken: 1.037 seconds, Fetched 1 row(s)

spark-sql> use ${db_name};
OK
Time taken: 1.697 seconds

so I think it may caused by this change: https://github.com/apache/spark/pull/8909 ( SPARK-10810 SPARK-10902 [SQL] Improve session management in SQL )

perhaps by calling hiveContext.newSession, the variables from sessionConf were not loaded into the new session? (https://github.com/apache/spark/pull/8909/files#diff-8f8b7f4172e8a07ff20a4dbbbcc57b1dR69)

Issue Links

• duplicates: SPARK-13768 Set hive conf failed use --hiveconf when beeline connect to thriftserver
• is duplicated by: SPARK-19927 SparkThriftServer2 can not get ''--hivevar" variables in spark 2.1
• relates to: SPARK-18086 Regression: Hive variables no longer work in Spark 2.0
• links to: [Github] Pull Request #15466 (wangyum)
• links to: [Github] Pull Request #17886 (wangyum)