[HIVE-9339] Optimize split grouping for CombineHiveInputFormat [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Spark
• Assignee: Jimmy Xiang
• Reporter: Xuefu Zhang
• Created: 12/Jan/15 03:11
• Updated: 03/Feb/15 17:21
• Resolved: 03/Feb/15 17:21

Description

It seems that split generation, especially in terms of grouping inputs, needs to be improved. For this, we may need cluster information. Because of this, we will first try to solve the problem for Spark.

As to cluster information, Spark doesn't provide an API (SPARK-5080). However, Spark doesn't have a listener API, with which Spark driver can get notifications about executor going up/down, task starting/finishing, etc. With this information, Spark client should be able to have a view of the current cluster image.

Spark developers mentioned that the listener can only be created after SparkContext is started, at which time, some executions may have already started and so the listener will miss some information. This can be fixed. File a JIRA with Spark project if necessary.