Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7552

Collect spark job statistic through spark metrics [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      MR/Tez use counters to collect job statistic information, while Spark does not use accumulator to do the same thing. Instead, Spark store task metrics information in TaskMetrics and send it back to scheduler. We could get spark job statistic information through combine all TaskMetrics with SparkListener.
      NO PRECOMMIT TESTS. This is for spark-branch only.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nyang Na Yang
                Reporter:
                chengxiang li Chengxiang Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: