[CARBONDATA-3448] Wrong results in preaggregate query with spark adaptive execution

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Ajantha Bhat
• Created: 21/Jun/19 05:33
• Updated: 26/Jun/19 05:37
• Resolved: 26/Jun/19 05:37

Description

problem: Wrong results in preaggregate query with spark adaptive execution

Spark2TestQueryExecutor.conf.set(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key, "true")

 

cause: For preaggreagate, segment info is set into threadLocal. when adaptive execution is called, spark is calling getInternalPartition in another thread where updated segment conf  is not set. Hence it is not using the updated segments.

 

solution: CarbonScanRdd is already having the sessionInfo, use it instead of taking session info from the current thread.

Issue Links

• links to: GitHub Pull Request #3303