DFSFileInfo should also have field to say if the file is underconstrction

• Key: HADOOP-3241
• Type: Bug
• Status: Closed
• Resolution: Won't Fix
• Priority: Major
• Assignee: Lohit Vijayarenu
• Reporter: Lohit Vijayarenu
• Affects Version/s: 0.18.0

Description

It would be good to have a flag which says if the DFSFileInfo object corresponds to a file under construction or not.

Comments

[Tsz Wo (Nicholas), SZE]

The flag is also useful for append. See HADOOP-1700.

[Lohit Vijayarenu]

This patch adds underConstruction flag to DFSFileInfo which is true only if its a file and is underConstruction. Have added a testcase to check this.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379977/HADOOP-3241-1.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2219/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2219/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2219/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2219/console

This message is automatically generated.

[Lohit Vijayarenu]

FileStats#equals() checks if the path contained in FileStatus are equal. if we consider underConstruction flag being dynamic just like length, then DFSFileInfo should also just check if path are same. So, for now, I am calling super.equals. This is also to get rid of findbugs warnings.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12379986/HADOOP-3241-2.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2221/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2221/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2221/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2221/console

This message is automatically generated.

[Doug Cutting]

The patch looks fine. But what is the use case?

[dhruba borthakur]

If my memory serves me right, the "append" design says that if a Reader retrieves a list of blocks from the namenode and the inode is marked "underConstruction", then the Reader can do something more intelligent then returning failure when the application tries to read from beyond the last known file size. This is more of an optimization and is not really required.

[Owen O'Malley]

If this isn't required, we should reject it.

The patch also has some strange code where the new methods just call their super methods. Is that necessary for some reason?

[dhruba borthakur]

I vote on closing this issue as a "won't fix" for now.

[Lohit Vijayarenu]

Sorry I missed updating this. When this JIRA was open, we thought it would be useful for Appends, but it might not be at this point of time. Closing this as wont fix