Hive
  1. Hive
  2. HIVE-2479

Log more Hadoop task counter values in the MapRedStats class.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      We should log more of the Hadoop task tracker counters in the MapRedStats class, in order to make them available to hooks and improve logging.

      We should make the counters that are logged configurable, so that if different Hadoop versions are used, different counters can be collected.

      1. HIVE-2479.4.patch.txt
        6 kB
        Kevin Wilfong
      2. HIVE-2479.3.patch.txt
        5 kB
        Kevin Wilfong
      3. HIVE-2479.2.patch.txt
        7 kB
        Kevin Wilfong
      4. HIVE-2479.1.patch.txt
        5 kB
        Kevin Wilfong

        Activity

        Kevin Wilfong created issue -
        Kevin Wilfong made changes -
        Field Original Value New Value
        Summary Log more Hadoop task counter values in the MapRedStats object. Log more Hadoop task counter values in the MapRedStats class.
        Description We should log more of the Hadoop task tracker counters in the MapRedStats object, in order to make them available to hooks and improve logging. Specifically these are the counters we should add:

            MAP_SPILL_CPU,
            MAP_SPILL_WALLCLOCK,
            MAP_SPILL_NUMBER,
            MAP_SPILL_BYTES,
            MAP_MEM_SORT_CPU,
            MAP_MEM_SORT_WALLCLOCK,
            MAP_MERGE_CPU,
            MAP_MERGE_WALLCLOCK,
            REDUCE_SHUFFLE_BYTES,
            REDUCE_COPY_WALLCLOCK,
            REDUCE_COPY_CPU,
            REDUCE_SORT_WALLCLOCK,
            REDUCE_SORT_CPU,
            MAP_TASK_WALLCLOCK,
            REDUCE_TASK_WALLCLOCK,
            MAP_INPUT_BYTES
        We should log more of the Hadoop task tracker counters in the MapRedStats class, in order to make them available to hooks and improve logging. Specifically these are the counters we should add:

            MAP_SPILL_CPU,
            MAP_SPILL_WALLCLOCK,
            MAP_SPILL_NUMBER,
            MAP_SPILL_BYTES,
            MAP_MEM_SORT_CPU,
            MAP_MEM_SORT_WALLCLOCK,
            MAP_MERGE_CPU,
            MAP_MERGE_WALLCLOCK,
            REDUCE_SHUFFLE_BYTES,
            REDUCE_COPY_WALLCLOCK,
            REDUCE_COPY_CPU,
            REDUCE_SORT_WALLCLOCK,
            REDUCE_SORT_CPU,
            MAP_TASK_WALLCLOCK,
            REDUCE_TASK_WALLCLOCK,
            MAP_INPUT_BYTES
        Kevin Wilfong made changes -
        Attachment HIVE-2479.1.patch.txt [ 12497665 ]
        Kevin Wilfong made changes -
        Attachment HIVE-2479.2.patch.txt [ 12497724 ]
        Kevin Wilfong made changes -
        Description We should log more of the Hadoop task tracker counters in the MapRedStats class, in order to make them available to hooks and improve logging. Specifically these are the counters we should add:

            MAP_SPILL_CPU,
            MAP_SPILL_WALLCLOCK,
            MAP_SPILL_NUMBER,
            MAP_SPILL_BYTES,
            MAP_MEM_SORT_CPU,
            MAP_MEM_SORT_WALLCLOCK,
            MAP_MERGE_CPU,
            MAP_MERGE_WALLCLOCK,
            REDUCE_SHUFFLE_BYTES,
            REDUCE_COPY_WALLCLOCK,
            REDUCE_COPY_CPU,
            REDUCE_SORT_WALLCLOCK,
            REDUCE_SORT_CPU,
            MAP_TASK_WALLCLOCK,
            REDUCE_TASK_WALLCLOCK,
            MAP_INPUT_BYTES
        We should log more of the Hadoop task tracker counters in the MapRedStats class, in order to make them available to hooks and improve logging.

        We should make the counters that are logged configurable, so that if different Hadoop versions are used, different counters can be collected.
        Kevin Wilfong made changes -
        Attachment HIVE-2479.3.patch.txt [ 12497861 ]
        Kevin Wilfong made changes -
        Attachment HIVE-2479.4.patch.txt [ 12497876 ]
        He Yongqiang made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Fix Version/s 0.8.0 [ 12316178 ]
        Carl Steinbach made changes -
        Fix Version/s 0.9.0 [ 12317742 ]
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Kevin Wilfong
            Reporter:
            Kevin Wilfong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development