Hadoop Common
  1. Hadoop Common
  2. HADOOP-1328

Hadoop Streaming needs to provide a way for the stream plugin to update global counters

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.
      Show
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.

      Description

      Sometimes, the stream plugin may want to create/update its own global counters.
      There is no way to do so currently.
      One possible way to enable that is to use the stderr output of the stream process.
      The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
      to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.

      1. hadoop-1328-v5.patch
        7 kB
        Tom White
      2. hadoop-1328-v4.patch
        7 kB
        Tom White
      3. hadoop-1328-v3.patch
        7 kB
        Tom White
      4. hadoop-1328-v2.patch
        7 kB
        Tom White
      5. hadoop-1328.patch
        7 kB
        Tom White

        Issue Links

          Activity

            People

            • Assignee:
              Tom White
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development