Hadoop Common
  1. Hadoop Common
  2. HADOOP-1328

Hadoop Streaming needs to provide a way for the stream plugin to update global counters

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.
      Show
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.

      Description

      Sometimes, the stream plugin may want to create/update its own global counters.
      There is no way to do so currently.
      One possible way to enable that is to use the stderr output of the stream process.
      The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
      to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.

      1. hadoop-1328.patch
        7 kB
        Tom White
      2. hadoop-1328-v2.patch
        7 kB
        Tom White
      3. hadoop-1328-v3.patch
        7 kB
        Tom White
      4. hadoop-1328-v4.patch
        7 kB
        Tom White
      5. hadoop-1328-v5.patch
        7 kB
        Tom White

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Tom!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Tom!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12383210/hadoop-1328-v5.patch
          against trunk revision 662813.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12383210/hadoop-1328-v5.patch against trunk revision 662813. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Synced with trunk.

          Show
          Tom White added a comment - Synced with trunk.
          Hide
          Tom White added a comment -

          Synced with trunk.

          Show
          Tom White added a comment - Synced with trunk.
          Hide
          Tom White added a comment -

          Updated to apply to trunk following changes in HADOOP-3089.

          Show
          Tom White added a comment - Updated to apply to trunk following changes in HADOOP-3089 .
          Hide
          Amareshwari Sriramadasu added a comment -

          +1

          Show
          Amareshwari Sriramadasu added a comment - +1
          Hide
          Tom White added a comment -

          I've generalized this slightly to support status updates by the same mechanism. There is a (configurable) prefix that allows you to access the reporter via stderr. To increment a counter:

          reporter:counter:<group>,<counter>,<amount>

          And to set the status:

          reporter:status:<message>

          Show
          Tom White added a comment - I've generalized this slightly to support status updates by the same mechanism. There is a (configurable) prefix that allows you to access the reporter via stderr. To increment a counter: reporter:counter:<group>,<counter>,<amount> And to set the status: reporter:status:<message>
          Hide
          Tom White added a comment -

          This patch adds support for counters in streaming. Scripts write lines to standard error of the form:

          counter:<group>,<counter>,<amount>

          For example, in python you might write

          sys.stderr.write("counter:counters,linecount,1\n")

          Show
          Tom White added a comment - This patch adds support for counters in streaming. Scripts write lines to standard error of the form: counter:<group>,<counter>,<amount> For example, in python you might write sys.stderr.write("counter:counters,linecount,1\n")

            People

            • Assignee:
              Tom White
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development