Design and Implement a Test Plan to support appends to HDFS files

• Key: HADOOP-2658
• Type: New Feature
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: dhruba borthakur
• Reporter: dhruba borthakur
• Component/s: test
• Fix Version/s: 0.19.0

Description

HADOOP-1700 describes the design of supporting appends to HDFS files. This feature needs extensive testing, especially because the design explicitly analyzes many failure scenarios. A detailed test plan and test cases are needed to make this feature a reality.

Comments

[Nigel Daley]

Dhruba, weren't you going to write a test plan?

[dhruba borthakur]

Test Plan in the works, but not ready yet. Will post in the next 4 days.

[stack]

+1 on the appendtestcase.doc (You might want to add author, date, and link to this issue Dhruba).

[dhruba borthakur]

Incorporated feedback from Stack. Thanks.

[Tsz Wo (Nicholas), SZE]

There are a few append-related tests failed occasionally. Linking the to this issue.

[Nigel Daley]

Attaching latest test plan in a non-proprietary file format.

[Tsz Wo (Nicholas), SZE]

20081008testplan.txt

I went thought each test in the test plan. We already have unit tests for some of the items. Some items need new unit tests. The remaining items need manual tests.

[Chris Douglas]

[ edit - sorry, wrong JIRA ]

[Chris Douglas]

Manual tests for items 4 and 14 passed, but 7 failed with the following exception:

08/10/13 04:53:20 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block blk_-2897952013639714624_1002java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readLong(DataInputStream.java:399)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2307)

08/10/13 04:53:20 WARN hdfs.DFSClient: Error Recovery for block blk_-2897952013639714624_1002 bad datanode[0] 67.195.209.61:50010
08/10/13 04:53:20 WARN hdfs.DFSClient: Error Recovery for block blk_-2897952013639714624_1002 in pipeline 67.195.209.61:50010, 67.195.209.109:50010: bad datanode 67.195.209.61:50010
08/10/13 04:53:20 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block blk_-2897952013639714624_1003java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:150)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:123)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readLong(DataInputStream.java:399)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2307)

08/10/13 04:53:20 WARN hdfs.DFSClient: Error Recovery for block blk_-2897952013639714624_1003 bad datanode[0] 67.195.209.109:50010

The remaining, good replica was lost.

[Tsz Wo (Nicholas), SZE]

TestFileAppend3_TC7.java: implemented TC7. Trunk cannot pass this test.

[Tsz Wo (Nicholas), SZE]

TC7 failed is because DataNode.getBlockMetaDataInfo(Block) does not aware the block is corrupted and it returns a BlockMetaDataInfo with the corrupted information.

[Tsz Wo (Nicholas), SZE]

The problem discovered by TC7 is now fixed by HADOOP-4423.

See 20081017testplan.txt for the test plan updates.

[Robert Chansler]

OK by Nigel and I!