Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9078

Hive should not submit second SparkTask while previous one has failed.[Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:

      Description

      hive> select n_name, c_name from nation, customer where nation.n_nationkey = customer.c_nationkey limit 10;
      Query ID = root_20141211135050_51e5ae15-49a3-4a46-826f-e27ee314ccb2
      Total jobs = 2
      Launching Job 1 out of 2
      In order to change the average load for a reducer (in bytes):
        set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
        set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
        set mapreduce.job.reduces=<number>
      Status: Failed
      Launching Job 2 out of 2
      In order to change the average load for a reducer (in bytes):
        set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
        set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
        set mapreduce.job.reduces=<number>
      Status: Failed
      OK
      Time taken: 68.53 seconds
      

      2 issue in the above CLI output.

      1. For a query which would be translated into multi SparkTask, is previous SparkTask failed, Hive should failed right away, the following SparkTask should not be submitted any more.
      2. Print failed info in Hive console while query failed.

      The correct CLI output while query failed:

      hive> select n_name, c_name from nation, customer where nation.n_nationkey = customer.c_nationkey limit 10;
      Query ID = root_20141211142929_ddb7f205-8422-44b4-96bd-96a1c9291895
      Total jobs = 2
      Launching Job 1 out of 2
      In order to change the average load for a reducer (in bytes):
        set hive.exec.reducers.bytes.per.reducer=<number>
      In order to limit the maximum number of reducers:
        set hive.exec.reducers.max=<number>
      In order to set a constant number of reducers:
        set mapreduce.job.reduces=<number>
      Status: Failed
      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
      

        Attachments

        1. HIVE-9078.1-spark.patch
          4 kB
          Xuefu Zhang
        2. HIVE-9078.2-spark.patch
          4 kB
          Xuefu Zhang
        3. HIVE-9078.3-spark.patch
          5 kB
          Chengxiang Li

          Issue Links

            Activity

              People

              • Assignee:
                chengxiang li Chengxiang Li
                Reporter:
                chengxiang li Chengxiang Li
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: