Hadoop Common
  1. Hadoop Common
  2. HADOOP-1328

Hadoop Streaming needs to provide a way for the stream plugin to update global counters

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.
      Show
      Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.

      Description

      Sometimes, the stream plugin may want to create/update its own global counters.
      There is no way to do so currently.
      One possible way to enable that is to use the stderr output of the stream process.
      The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
      to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.

      1. hadoop-1328-v5.patch
        7 kB
        Tom White
      2. hadoop-1328-v4.patch
        7 kB
        Tom White
      3. hadoop-1328-v3.patch
        7 kB
        Tom White
      4. hadoop-1328-v2.patch
        7 kB
        Tom White
      5. hadoop-1328.patch
        7 kB
        Tom White

        Issue Links

          Activity

          Runping Qi created issue -
          Sameer Paranjpye made changes -
          Field Original Value New Value
          Component/s contrib/streaming [ 12310972 ]
          Description
          Sometimes, the stream plugin may want to create/update its own global counters.
          There is no way to do so currently.
          One possible way to enable that is to use the stderr output of the stream process.
          The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
          to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
          Sometimes, the stream plugin may want to create/update its own global counters.
          There is no way to do so currently.
          One possible way to enable that is to use the stderr output of the stream process.
          The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
          to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
          Assignee Owen O'Malley [ owen.omalley ]
          FM FLOCH made changes -
          Link This issue is blocked by HADOOP-1915 [ HADOOP-1915 ]
          Owen O'Malley made changes -
          Assignee Owen O'Malley [ owen.omalley ] Christopher Zimmerman [ zim ]
          Hide
          Tom White added a comment -

          This patch adds support for counters in streaming. Scripts write lines to standard error of the form:

          counter:<group>,<counter>,<amount>

          For example, in python you might write

          sys.stderr.write("counter:counters,linecount,1\n")

          Show
          Tom White added a comment - This patch adds support for counters in streaming. Scripts write lines to standard error of the form: counter:<group>,<counter>,<amount> For example, in python you might write sys.stderr.write("counter:counters,linecount,1\n")
          Tom White made changes -
          Attachment hadoop-1328.patch [ 12377792 ]
          Hide
          Tom White added a comment -

          I've generalized this slightly to support status updates by the same mechanism. There is a (configurable) prefix that allows you to access the reporter via stderr. To increment a counter:

          reporter:counter:<group>,<counter>,<amount>

          And to set the status:

          reporter:status:<message>

          Show
          Tom White added a comment - I've generalized this slightly to support status updates by the same mechanism. There is a (configurable) prefix that allows you to access the reporter via stderr. To increment a counter: reporter:counter:<group>,<counter>,<amount> And to set the status: reporter:status:<message>
          Tom White made changes -
          Attachment hadoop-1328-v2.patch [ 12378493 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          +1

          Show
          Amareshwari Sriramadasu added a comment - +1
          Hide
          Tom White added a comment -

          Updated to apply to trunk following changes in HADOOP-3089.

          Show
          Tom White added a comment - Updated to apply to trunk following changes in HADOOP-3089 .
          Tom White made changes -
          Attachment hadoop-1328-v3.patch [ 12379378 ]
          Hide
          Tom White added a comment -

          Synced with trunk.

          Show
          Tom White added a comment - Synced with trunk.
          Tom White made changes -
          Attachment hadoop-1328-v4.patch [ 12380283 ]
          Hide
          Tom White added a comment -

          Synced with trunk.

          Show
          Tom White added a comment - Synced with trunk.
          Tom White made changes -
          Attachment hadoop-1328-v5.patch [ 12383210 ]
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Christopher Zimmerman [ zim ] Tom White [ tomwhite ]
          Hadoop Flags [Reviewed]
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tom White made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12383210/hadoop-1328-v5.patch
          against trunk revision 662813.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12383210/hadoop-1328-v5.patch against trunk revision 662813. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/console This message is automatically generated.
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Tom!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Tom!
          Owen O'Malley made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Devaraj Das made changes -
          Release Note Introduces a way for the streaming process to update global counters and status. A streaming process can use the stderr to emit counter information. reporter:counter:<group>,<counter>,<amount> can be used to update a counter. And to set a status, reporter:status:<message> can be used.
          Robert Chansler made changes -
          Release Note Introduces a way for the streaming process to update global counters and status. A streaming process can use the stderr to emit counter information. reporter:counter:<group>,<counter>,<amount> can be used to update a counter. And to set a status, reporter:status:<message> can be used. Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use "reporter:counter:<group>,<counter>,<amount> " to update a counter. Use "reporter:status:<message>" to update status.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s contrib/streaming [ 12310972 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          1d 4h 1 Tom White 03/Jun/08 14:38
          Open Open Patch Available Patch Available
          394d 12h 2 Tom White 03/Jun/08 14:38
          Patch Available Patch Available Resolved Resolved
          1d 6h 43m 1 Owen O'Malley 04/Jun/08 21:21
          Resolved Resolved Closed Closed
          78d 23h 28m 1 Nigel Daley 22/Aug/08 20:50

            People

            • Assignee:
              Tom White
              Reporter:
              Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development