[SPARK-23304] Spark SQL coalesce() against hive not working

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Assignee: Xiao Li
• Reporter: Thomas Graves
• Created: 01/Feb/18 18:20
• Updated: 06/Feb/18 04:01
• Resolved: 02/Feb/18 14:54

Description

The query below seems to ignore the coalesce. This is running spark 2.2 or spark 2.3 against hive, which is reading orc:

 
Query:
spark.sql("SELECT COUNT(DISTINCT(something)) FROM sometable WHERE dt >= '20170301' AND dt <= '20170331' AND something IS NOT NULL").coalesce(160000).show()