Hadoop Streaming needs to provide a way for the stream plugin to update global counters

• Key: HADOOP-1328
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Tom White
• Reporter: Runping Qi
• Fix Version/s: 0.18.0

Description

Sometimes, the stream plugin may want to create/update its own global counters.
There is no way to do so currently.
One possible way to enable that is to use the stderr output of the stream process.
The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.

Comments

[Tom White]

This patch adds support for counters in streaming. Scripts write lines to standard error of the form:

counter:<group>,<counter>,<amount>

For example, in python you might write

sys.stderr.write("counter:counters,linecount,1\n")

[Tom White]

I've generalized this slightly to support status updates by the same mechanism. There is a (configurable) prefix that allows you to access the reporter via stderr. To increment a counter:

reporter:counter:<group>,<counter>,<amount>

And to set the status:

reporter:status:<message>

[Amareshwari Sriramadasu]

+1

[Tom White]

Updated to apply to trunk following changes in HADOOP-3089.

[Tom White]

Synced with trunk.

[Tom White]

Synced with trunk.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383210/hadoop-1328-v5.patch
against trunk revision 662813.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 9 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2549/console

This message is automatically generated.

[Owen O'Malley]

I just committed this. Thanks, Tom!