Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9339

Optimize split grouping for CombineHiveInputFormat [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      It seems that split generation, especially in terms of grouping inputs, needs to be improved. For this, we may need cluster information. Because of this, we will first try to solve the problem for Spark.

      As to cluster information, Spark doesn't provide an API (SPARK-5080). However, Spark doesn't have a listener API, with which Spark driver can get notifications about executor going up/down, task starting/finishing, etc. With this information, Spark client should be able to have a view of the current cluster image.

      Spark developers mentioned that the listener can only be created after SparkContext is started, at which time, some executions may have already started and so the listener will miss some information. This can be fixed. File a JIRA with Spark project if necessary.

        Attachments

          Activity

            People

            • Assignee:
              jxiang Jimmy Xiang
              Reporter:
              xuefuz Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: