|
Attaching a patch which addes FSNamesystem status metrics.
Since these are not timevarying int or timevaying rate. I use MetricsLongValue similar to MetricsIntValue using their set and get methods. The metrics are recorded as FSNamesystem record and list these
Tested this using FileContext to log these while FSNamesystem was reporting and I could see the values being updated +1 Patch looks fine. I have no idea if this has real penalty on namenode performance. Mostly it does not affect. Multiple simple integer and long operation in the critical paths are replaced by methods that synchronize on a different object.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381301/HADOOP-3058.patch  against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included -1. The patch doesn't appear to include any new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2365/testReport/ This message is automatically generated. FSNameSystem has 2 constructors and registerMBean was being called only one place. This was causing NPE, fixed it in the updated patch
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381365/HADOOP-3058-2.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included -1. The patch doesn't appear to include any new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. release audit +1. The applied patch does not generate any new release audit warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2387/testReport/ This message is automatically generated. Yes, I also agree that these adds more operations. The metrics frequently updated are
Few metrics are replaced by updating a global variables regarding the DFS capacity. These were updated on each heart beat once, which should be fine. I would think the heartBeat variables will be updated thousands of times every second. I think better approach would be to update the heavy weight metric variables only inside FSNamesystemMetrics.doUpdates() that gets called every 5 seconds or so. This way these stats become pretty much free and also sets good precedence for new metrics. Thanks Rahgu, I have attached an updated patch in which we maintain local counters in FSNamesystem as earlier. FSNamesystemMetrics object is updated only during invoking of doUpdates()
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381578/HADOOP-3058-3.patch against trunk revision 654128. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2418/testReport/ This message is automatically generated. I just committed this. Thanks Lohit!
Integrated in Hadoop-trunk #484 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/484/
) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Closed
Minor
Hadoop DFS to report more replication metrics
In addition to the above, I forgot to ask for the most basic stats. It's very important that the NN send metrics on the number of files and blocks in the system, so that we can trend these over time. Including the number of directories would be a bonus.