|
The attached file fixes this problem (using svn diff against trunk); it allocates a full-sized array, and later truncates the array as necessary using System.arraycopy.
However, the whole thing smells of over-optimization. I would think that a better approach would be to rewrite the entire block: Convert everything to sets, and do the calculations based on the intersection of the sets. The initial problem was due to the fact that there were corrupt blocks that aren't in the block map. I'm not sure if this is an allowed situation (i.e., is this situation due to a bug elsewhere in the code?) I suggest to do the following improvements:
Hey Nicholas,
Per your second point: That would have been the natural thing to do. However, I thought that, because everything was written in terms of arrays, this might be a section where performance is critical. If performance is not critical in this section, then rewriting things as an ArrayList/Set would be appropriate. Do you know if it is or not? Brian > However, I thought that, because everything was written in terms of arrays, this might be a section where performance is critical.
You made a good point here. It will affect the performance if we use ArrayList. When I take a look at the codes again, it seems that there are bugs somewhere. Otherwise, we won't have ArrayIndexOutOfBoundsException. So we better fix the bugs. > it appears that the error happens when there are corrupt entries which are not in the blocksMap, but they are in the corruptReplicas object.
The solutions discussed avoid the exception to be thrown, but it does not fix the above fundamental problem. Hi Brian, the debugging output that you provided is very informative. Can you still reproduce the problem? Can you print which replicas are in curruptReplicasMap?
Hey Hairong,
Unfortunately, the admins moved the namenode to a more permanent home and clobbered the existing logfiles in the process. (And I fixed the corrupt blocks: I didn't want to live with a problematic file system for too long!) The code printed out the replicas in the blocksMap (node 10, 145, 117) and the corrupt entries in the corruptReplicas vairable (node16). The existing code calculates that 3 - 1 = 2 replicas must be good (this is a mistake as corruptReplicas is not a subset of blocksMap); however, when it starts to populate the machineSet, it only gets as far as nodes 10 and 145, then throws the exception on 117. You're right - there's possibly an underlying problem here which this patches a symptom of. However, it still is a useful thing to fix: it is rather painful to go through filesystem cleanup when fsck dies instantly. I believe there log message was something like this one: 2008-10-07 05:04:24,021 INFO org.apache.hadoop.dfs.StateChange: BLOCK NameSystem.markBlockAsCorrupt: block blk_-5420894356244363410_2169 could not be marked as corrupt as it does not exists in blocksMap I think if a data node reports it has a corrupt block, then it gets added to the corrupt map and not the blocksMap. I took a peek at the underlying issue, and wasn't able to make much progress - an expert on FSNamesystem will be needed to find the underlying problem. Brian PS: I looked at the patch again - silly me leaked several changes I have made into this patch; please disregard any changes to files that are NOT FSNamesystem.java Thanks Brian! The information you reported is very helpful. I checked the code, I saw two scenarios that might cause the inconsistency: a replica is in corruptedBlockMap but not in blockMap.
1. In FSNamesystem.markBlockAsCorrupted, a replica is added to corruptedBlocksMap even if it is not in the blocksMap; This is a patch that fixes the two inconsistency scenarios.
Hairong also mentioned that she conducted a test where the fault was observed without the patch, and the fault was not observed with the patch. Incidentally, the fault is only observed when the intended replication factor for the block is greater than one. Otherwise the block is reported as missing rather than having a bad replicas.
This new patch adds two more changes:
1. Prevent ArrayIndexOutOfBoundException being thrown even when inconsistency happens. Instead it prints a warning in the log. 2. Allow a file's blocks to exit the inconsistency state by increasing the file's replication factor. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391889/arrayIndexOutOfBounds1.patch  against trunk revision 703508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3439/testReport/ This message is automatically generated. This patch fixes a bug in markBlockAsCorrupt:
"updateNeededReplications(blk, 0, 1);" should be "updateNeededReplications(blk, -1, 0);" because replication factor does not change but the current number of replicas is decremented. This patch incorporates Konstantin's comments.
Some comments:
This patch incorporates Konstantin's comments.
+1 other than that. +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391911/arrayIndexOutOfBounds3.patch against trunk revision 703508. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3441/testReport/ This message is automatically generated. I just committed this.
Hi Konstantin, the patch that I uploaded at 4:20pm on 10/10 addressed all your review comments including last two that you commented after 5:00pm. That's also the one that I committed this morning. But looks like that I accidentally deleted it when I was managing the attachments. I just uploaded the one that I committed. Please double check. Here is the patch for branch 18.
Integrated in Hadoop-trunk #634 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/634/
)
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Blocker
ArrayIndexOutOfBoundsException during fsck
Adding in some debugging statements, it appears that the error happens when there are corrupt entries which are not in the blocksMap, but they are in the corruptReplicas object.
Here's my debug statement output:
2008-10-06 10:57:17,987 ERROR org.apache.hadoop.fs.FSNamesystem: Internal error in calculating corrupt block locations for blk_-8711032927274230751_1834
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: There should be 2 corrupt replicas; there are 3 known replicas. There are 1 known corrupt replicas.
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: We believe the value of blockCorrupt is false. Current counter value: 2
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * Corrupt entry: 172.16.1.16:50010
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * machineSet entry: 172.16.1.10:50010
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * machineSet entry: 172.16.1.145:50010
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * block replica entry: 172.16.1.10:50010
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * block replica entry: 172.16.1.145:50010
2008-10-06 10:57:17,988 ERROR org.apache.hadoop.fs.FSNamesystem: * block replica entry: 172.16.1.117:50010
So, nodes 10, 145, and 117 hold good replicas; 16 held a bad one. However, because the allocated array size is (# of replicas) - (# of bad replicas), machineSet is allocated a size of 2, but we try to write 3 elements into it.