[#HADOOP-3168] reduce amount of logging in hadoop streaming

Hadoop Common

reduce amount of logging in hadoop streaming

Created: 03/Apr/08 11:37 PM Updated: 08/Jul/09 05:05 PM

Return to search

Component/s:

None

Affects Version/s:

0.15.3, 0.16.0, 0.16.1, 0.16.2

Fix Version/s:

0.17.0

Time Tracking:

Not Specified

File Attachments:

	File Name	Date Attached	Attached By	Size
	patch.txt	2008-04-04 12:24 AM	Zheng Shao	0.6 kB

Hadoop Flags:	Reviewed
Release Note:	Decreases the frequency of logging from streaming from every 100 records to every 10,000 records.
Resolution Date:	04/Apr/08 08:39 PM

Description

« Hide

logs every 100 records:

void maybeLogRecord() {
if (numRecRead_ >= nextRecReadLog_) { String info = numRecInfo(); logprintln(info); logflush(); //nextRecReadLog_ *= 10; nextRecReadLog_ += 100; }
}

ouch.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Zheng Shao added a comment - 04/Apr/08 12:23 AM

Output less and less log messages until we reach one log message per 100,000 records.

[ Permlink | « Hide ]

Hadoop QA added a comment - 04/Apr/08 01:26 PM

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379339/patch.txt
against trunk revision 643282.

@author +1. The patch does not contain any @author tags.

tests included -1. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2159/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2159/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2159/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2159/console

This message is automatically generated.

[ Permlink | « Hide ]

Owen O'Malley added a comment - 04/Apr/08 04:18 PM

[ Permlink | « Hide ]

Owen O'Malley added a comment - 04/Apr/08 08:39 PM

I just committed this. Thanks, Zheng!

[ Permlink | « Hide ]

Hudson added a comment - 05/Apr/08 12:13 PM

Integrated in Hadoop-trunk #451 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/)