[HADOOP-83] infinite retries accessing a missing block

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.1.0
• Fix Version/s: 0.1.0
• Assignee: Konstantin Shvachko
• Reporter: Yoram Arnon
• Created: 16/Mar/06 02:59
• Updated: 08/Jul/09 16:41
• Resolved: 24/Mar/06 02:13

Description

A file in the DFS got corrupted - the reason for that is unknown, but might be justified.

when accessing the file, I get an infinite stream of error messages from the client - attached below.
The client aparently increments an error counter, but never checks it.

Correct behaviour is for the client to retry a few times, then abort.

060315 105436 No node available for block blk_2690692619196463439
060315 105436 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105446 No node available for block blk_2690692619196463439
060315 105446 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105456 No node available for block blk_2690692619196463439
060315 105456 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105506 No node available for block blk_2690692619196463439
060315 105506 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105516 No node available for block blk_2690692619196463439
060315 105516 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105526 No node available for block blk_2690692619196463439
060315 105526 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105536 No node available for block blk_2690692619196463439
060315 105536 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105546 No node available for block blk_2690692619196463439
060315 105546 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105556 No node available for block blk_2690692619196463439
060315 105556 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105606 No node available for block blk_2690692619196463439
060315 105606 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105616 No node available for block blk_2690692619196463439
060315 105616 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105626 No node available for block blk_2690692619196463439
060315 105626 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105636 No node available for block blk_2690692619196463439
060315 105636 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105646 No node available for block blk_2690692619196463439
060315 105646 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105657 No node available for block blk_2690692619196463439
060315 105657 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105707 No node available for block blk_2690692619196463439
060315 105707 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105717 No node available for block blk_2690692619196463439
060315 105717 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105727 No node available for block blk_2690692619196463439
060315 105727 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105737 No node available for block blk_2690692619196463439
060315 105737 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105747 No node available for block blk_2690692619196463439
060315 105747 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105757 No node available for block blk_2690692619196463439
060315 105757 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105807 No node available for block blk_2690692619196463439
060315 105807 Could not obtain block from any node: java.io.IOException: No live nodes contain current block
060315 105817 No node available for block blk_2690692619196463439

Issue Links

• duplicates: HADOOP-35 Files missing chunks can cause mapred runs to get stuck
• incorporates: HADOOP-85 a single client stuck in a loop blocks all clients on same machine
• relates to: HADOOP-1911 infinite loop in dfs -cat command.