Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 0.18.2, 0.19.0
-
Fix Version/s: 0.18.2
-
Component/s: metrics
-
Labels:None
-
Environment:
Red Hat Enterprise Linux AS release 4 (Nahant Update 5), Java 1.6.
-
Hadoop Flags:Incompatible change, Reviewed
-
Release Note:Changed bytes_read, bytes_written to type long to prevent metrics overflow.
Description
bytes_read, and bytes_written metrics are using int (MetricsTimeVaryingInt) as counter. This type is too small to store the bytes_read and bytes_written metrics. Recommend to change this to long (metricsLongValue).
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Patch to change bytes_read and bytes_written metrics from type Integer to type long.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390702/HADOOP-4228-trunk.patch
against trunk revision 697306.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3349/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3349/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3349/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3349/console
This message is automatically generated.
The failed test has no relationship to this patch. There is probably something else committed to trunk which caused Balancer test failure.
The SimulatedDataNodeMetrics seems quite dummy, which is introduced for only the testing purpose.
A side note: DataNodeStatics and DataNodeStaticsMBean collect bytesRead but not bytesWritten. They should collect metrics bytesWritten as well.
This patch has the following changes:
1. DataNode's metrics bytesRead & bytesWritten are changed to be long;
2. Add getBytesWritten method to DataNodesStaticsticsMBean;
3. Add a unit test to see if DataNodeMetrics can handle long number of bytes written.
+1
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391383/metricsLong.patch
against trunk revision 701476.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3431/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3431/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3431/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3431/console
This message is automatically generated.
This is a patch for branch 18.
This is a patch for the trunk. It removes the change to DFSTestUtil that caused the failed tests.
Changing the type to Long breaks Ganglia metrics reporting (there's a static mapping of Java type -> Ganglia type; there's no current Long type, meaning the map returns null and causes a NPE).
Ganglia has no Long type - perhaps a solution would be to map Longs in Ganglia to a float?
See also:
HADOOP-4137
HADOOP-3422
> Ganglia has no Long type - perhaps a solution would be to map Longs in Ganglia to a float?
Short of supporting the Long data type, this seems like a reasonable thing to do.
Brian, would you mind coming up with the patch for the Ganglia context?
Thank Brian for submitting a patch to HADOOP-4137 for handling Long type in Ganglia context. I am resubmitting my patch to hudson.
+1 for Hairong's patch. The patch provides real test cases.
Resubmit the patch since the hudson patch process is back.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391556/metricsLong1.patch
against trunk revision 703609.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3442/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3442/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3442/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3442/console
This message is automatically generated.
I just committed this. Thanks Eric!
Integrated in Hadoop-trunk #632 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/632/)
. The type of dfs datanoe metrics, bytes_read and bytes_written, is changed to be long. Contributed by Eric Yang and Hairong Kuang.
Patch to change metrics type from Integer to Long.