Disable append

• Key: HADOOP-5224
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Unassigned
• Reporter: Nigel Daley
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.1

Description

As discussed on core-dev@ (http://www.nabble.com/Hadoop-0.19.1-td21739202.html) we will disable append API in Hadoop 0.19.1 by throwing UnsupportedOperationException from FileSystem.append(...) methods.

Does append in libhdfs (HADOOP-4494) need attention too?

Comments

[Jim Kellerman]

Where does this leave HADOOP-4379?

The testing I have done to date indicates that sync() does, in fact, work. See HBASE-1155

[Raghu Angadi]

Jim,

btw, this is assigned to me just to provide a patch. The decision is not mine. I will let others reply to the above valid concern.

[Robert Chansler]

This particular item turns off append() at the interface without affecting the implementation or normal behavior of sync().
A related item, Hadoop-5225, disables some of the recovery code associated with sync in the case of data node failure.

The use of either of these facilities in 0.19.0 is dangerous to the integrity of the file system. The goal is to provide 0.19.1 with the same file system integrity as 0.18.3. Further development of append and sync recovery can then continue.

[Raghu Angadi]

patch for review. The only real change is that DistributedFileSystem.append() throws UnsupportedOperationException. Rest of the changes are for disabling tests that depend on HDFS append.

[Raghu Angadi]

'ant test-contrib' passes as well.

[Raghu Angadi]

'ant test-libhdfs' passes as well.

ant test-patch :

     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.

[Konstantin Shvachko]

Looks like TestStickyBit uses append(). Something tells me it should have failed.

[Raghu Angadi]

correct. But TestStickyBit is not in 0.19.

[Konstantin Shvachko]

True. +1

[Raghu Angadi]

I just committed committed this to 0.19 (only).

[dhruba borthakur]

Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps. There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true.

I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue.

[dhruba borthakur]

Making the config parameter dfs.support.append to "false" by default might allow Jim Kellerman to continue testing his Hbase code with 0.19.1 (till these get resolved in later releases).

[Raghu Angadi]

I think I shouldn't have been assigned this jira in the first place.

[dhruba borthakur]

Hi Raghu, I uploaded a patch where the only change is to set "dfs.support.append" to false. Does this look ok to you?

[Nigel Daley]

Hi folks, please pardon me for re-opening this issue. I am wondering if there is a better way to disable "appends" for 0.19.1 We coudld set the config parameter dfs.support.append to false in hadoop-defaults.xml. This means that a default installation of hadoop will not support "appends". However, for people that are testing their apps (not in production), it gives them an easy way to test their apps.

Dhruba, folks who want to test with append can use 0.19.0.

There could be big-and-large warning message associated with the dfs.support.append parameter warning people to the fact that there could be data corruption if they set that flag to true.

The uploaded patch creates a dfs.support.append config entry but doesn't seem to use it in the append method to affect it's behavior. What am I missing?

I am re-opening this issue because I am not sure whether this approach was considered. If it was considered and was shot down, please let me know why and u can close this issue.

I think this should be a new Jira that could be fixed in 0.19.2. If you agree, please close this and open a new Jira.

[dhruba borthakur]

Nigel, thanks for the suggestion. I am closing this one.

[Raghu Angadi]

> 'ant test-libhdfs' passes as well.

my bad. I had reverted the patch by mistake before running 'ant test-libhdfs'. So the patch I attached is incomplete since it does not remove the libhdfs append test. Please disregard it.

[Raghu Angadi]

libhdfs.patch : disables append tests in libhdfs.

The command ' ant -Dcompile.c++=true -Dlibhdfs=true clean compile test-libhdfs ' fails with current 0.19 and passes with this patch applied.

I will commit this correction soon today.

[Raghu Angadi]

I just committed libhdfs.patch to 0.19 (only).