[SPARK-18086] Regression: Hive variables no longer work in Spark 2.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: SQL
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 25/Oct/16 00:35
• Updated: 07/May/17 04:07
• Resolved: 08/Nov/16 01:35

Description

The behavior of variables in the SQL shell has changed from 1.6 to 2.0. Specifically, --hivevar name=value and SET hivevar:name=value no longer work. Queries that worked correctly in 1.6 will either fail or produce unexpected results in 2.0 so I think this is a regression that should be addressed.

Hive and Spark 1.6 work like this:
1. Command-line args --hiveconf and --hivevar can be used to set session properties. --hiveconf properties are added to the Hadoop Configuration.
2. SET adds a Hive Configuration property, SET hivevar:<name>=<value> adds a Hive var.
3. Hive vars can be substituted into queries by name, and Configuration properties can be substituted using hiveconf:name.

In 2.0, hiveconf, sparkconf, and conf variable prefixes are all removed, then the value in SQLConf for the rest of the key is returned. SET adds properties to the session config and (according to a comment) the Hadoop configuration "during I/O".

Hive and Spark 1.6.1 behavior

[user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
spark-sql> select "${hiveconf:test.conf}";
1
spark-sql> select "${test.conf}";
${test.conf}
spark-sql> select "${hivevar:test.var}";
2
spark-sql> select "${test.var}";
2
spark-sql> set test.set=3;
SET test.set=3
spark-sql> select "${test.set}"
"${test.set}"
spark-sql> select "${hivevar:test.set}"
"${hivevar:test.set}"
spark-sql> select "${hiveconf:test.set}"
3
spark-sql> set hivevar:test.setvar=4;
SET hivevar:test.setvar=4
spark-sql> select "${hivevar:test.setvar}";
4
spark-sql> select "${test.setvar}";
4

Spark 2.0.0 behavior

[user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
spark-sql> select "${hiveconf:test.conf}";
1
spark-sql> select "${test.conf}";
1
spark-sql> select "${hivevar:test.var}";
${hivevar:test.var}
spark-sql> select "${test.var}";
${test.var}
spark-sql> set test.set=3;
test.set        3
spark-sql> select "${test.set}";
3
spark-sql> set hivevar:test.setvar=4;
hivevar:test.setvar      4
spark-sql> select "${hivevar:test.setvar}";
4
spark-sql> select "${test.setvar}";
${test.setvar}

Issue Links

• is related to: SPARK-13983 HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version (both multi-session and single session)
• links to: [Github] Pull Request #15738 (rdblue)