Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15237

Propagate Spark job failure to Hive

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.3.0
    • Spark
    • None

    Description

      If a Spark job failed for some reason, Hive doesn't get any additional error message, which makes it very hard for user to figure out why. Here is an example:

      Status: Running (Hive on Spark job[0])
      Job Progress Format
      CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount [StageCost]
      2016-11-17 21:32:53,134	Stage-0_0: 0/23	Stage-1_0: 0/28	
      2016-11-17 21:32:55,156	Stage-0_0: 0(+1)/23	Stage-1_0: 0/28	
      2016-11-17 21:32:57,167	Stage-0_0: 0(+3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:00,216	Stage-0_0: 0(+3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:03,251	Stage-0_0: 0(+3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:06,286	Stage-0_0: 0(+4)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:09,308	Stage-0_0: 0(+2,-3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:12,332	Stage-0_0: 0(+2,-3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:13,338	Stage-0_0: 0(+21,-3)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:15,349	Stage-0_0: 0(+21,-5)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:16,358	Stage-0_0: 0(+18,-8)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:19,373	Stage-0_0: 0(+21,-8)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:22,400	Stage-0_0: 0(+18,-14)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:23,404	Stage-0_0: 0(+15,-20)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:24,408	Stage-0_0: 0(+12,-23)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:25,417	Stage-0_0: 0(+9,-26)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:26,420	Stage-0_0: 0(+12,-26)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:28,427	Stage-0_0: 0(+9,-29)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:29,432	Stage-0_0: 0(+12,-29)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:31,444	Stage-0_0: 0(+18,-29)/23	Stage-1_0: 0/28	
      2016-11-17 21:33:34,464	Stage-0_0: 0(+18,-29)/23	Stage-1_0: 0/28	
      Status: Failed
      FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
      

      It would be better if we can propagate Spark error to Hive.

      Attachments

        1. HIVE-15237.2.patch
          5 kB
          Rui Li
        2. HIVE-15237.2.patch
          5 kB
          Rui Li
        3. HIVE-15237.patch
          7 kB
          Xuefu Zhang

        Activity

          People

            lirui Rui Li
            xuefuz Xuefu Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: