|
[
Permlink
| « Hide
]
Lohit Vijayarenu added a comment - 11/Apr/08 05:06 PM
Attached patch handles this case. This patch ignores open files by default but lists them as total number of open files. It provides an option '-open' when used will print the open files.
1. The ClientProtocol # has to be bumped because LocatedBlocks is used in a few RPCs. Please check if NamenodeProtocol uses it too.
2. Maybe it is better to add a new field to DFSFileInfo to say whether a file is opened for writing. The open/close characteristics is more related to a file rather than a block. This can replace the new field that you added to LocatedBlocks. 3. The summary prints total file and total open files in two separate lines. One confusion is whether the total file count includes the open files or not. It might be better to put these two in the same line 4. The "open" parameter to fsck might confuse people to wonder whether these files are opened for reading or writing. It would be worthwhile to come up with a different name of this command line parameter. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379934/HADOOP-2703-1.patch  against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2210/testReport/ This message is automatically generated. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379934/HADOOP-2703-1.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2211/testReport/ This message is automatically generated. Thanks Dhruba. I have incorporated changes suggested by you. This patch changes -open to -openforwrite, changes the way output is displayed and also bumped ClientProtocol versionID.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379966/HADOOP-2703-2.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2225/testReport/ This message is automatically generated. This updated patch adds few more details. When hadoop fsck is run without any options it along with total number of open files, it also lists total number of blocks owned by those open files and also its size if it is greater than 0. When -openforwrite option is used, all of them are added up and reported as total blocks.
Sample output when default fsck is invoked while a file is being written [lohit@ hadoop-core-trunk]$ ./bin/hadoop fsck / ...Status: HEALTHY Total size: 420912 B Total dirs: 5 Total files: 3 (Files currently being written: 1) Total blocks: 3 (avg. block size 140304 B) (Total open file blocks: 1) Minimally replicated blocks: 3 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 The filesystem under path '/' is HEALTHY [lohit@ hadoop-core-trunk]$ Attached slightly modified patch with changes to messages printed. Here is the summary.
Here is sample output of hadoop fsck / [lohit@ hadoop-core-trunk]$ ./bin/hadoop fsck / ....Status: HEALTHY Total size: 2148912 B Total dirs: 5 Total files: 4 (Files currently being written: 1) Total blocks (validated): 4 (avg. block size 537228 B) (Total open file blocks (not validated): 1) Minimally replicated blocks: 4 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 The filesystem under path '/' is HEALTHY [lohit@ hadoop-core-trunk]$ -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380220/HADOOP-2703-4.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2248/testReport/ This message is automatically generated. Had made error while printing out message which was caught by my testcase itself.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380257/HADOOP-2703-5.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 3 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2254/testReport/ This message is automatically generated. I just committed this. Thanks Lohit!
Integrated in Hadoop-trunk #463 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/463/
) Integrated in Hadoop-trunk #523 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/523/
) |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Minor
New files under lease (before close) still shows up as MISSING files/blocks in fsck
