[SPARK-6757] spark.sql.shuffle.partitions is global, not per connection

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: David Ross
• Created: 08/Apr/15 00:16
• Updated: 08/Apr/15 19:19
• Resolved: 08/Apr/15 19:19

Description

We are trying to use the spark.sql.shuffle.partitions parameter to handle large queries differently from smaller queries. We expected that this parameter would be respected per connection, but it seems to be global.

For example, in try this in two separate JDBC connections:

Connection 1:

SET spark.sql.shuffle.partitions=10;
SELECT * FROM some_table;

The correct number 10 was used.

Connection 2:

SET spark.sql.shuffle.partitions=100;
SELECT * FROM some_table;

The correct number 100 was used.

Back to connection 1:

SELECT * FROM some_table;

We expected the number 10 to be used but 100 is used.

Issue Links

• duplicates: SPARK-2087 Clean Multi-user semantics for thrift JDBC/ODBC server.