Ah turns out I can read the file, just not the block on the local machine (the "Try reading via the datanode on" didn't lie as I originally thought).

so is this still a bug or can it be closed as invalid?

Suresh, I changed the title to reflect what I think is really the issue, and yes this is a bad bug.

Another failure mode is when the block is done being written to (block unrelated with the one in the description):

2012-08-10 00:17:02,804 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_5190712822654892045_134607537 from cache because local file /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 could not be opened.
2012-08-10 00:17:02,804 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_5190712822654892045_134607537 on local machine java.io.FileNotFoundException: /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 (No such file or directory)

I still don't understand what's the length mismatch here, but one way we can fix this is by not using the BlockReaderLocal when the block is being written to. This is a 2 lines change which I'm attaching here. I tested it on a single node and it works, I'm about to deploy on a few clusters.

I guess I was tired when I did the other patch and missed the refactoring opportunity, attaching new version against branch-1.

We've been running with my patch in production for 2 weeks now on a few hundred nodes with HBase doing reads on local files that are still being written to. Everything looks good.

You have patch for trunk J-D so can run it by hadoopqa?

Attaching trunk's version. It's been a while since I've been in trunk so apologies if I did anything wrong.

It seems the input stream was moved out of DFSClient so I had to do some massaging. Basically I'm trying to keep shouldTryShortCircuitRead the same way I changed it for branch-1 but now the check is done in DFSInputStream.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12542461/HDFS-2757-trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.TestHftpDelegationToken

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//console

This message is automatically generated.

TestHftpDelegationToken fails even without my patch.

+1 looks good, thanks JD!

I thought both findbugs warnings were bogus but actually the 2nd one is due to this patch. The access to locatedBlocks needs to be synchronized, in practice it should be fine, but findbugs complains because all other accesses are synchronized. I pulled your check out to a private synchronized method. Updated patch attached.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12544365/hdfs-2757.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//console

This message is automatically generated.

+1 on Eli patch

Integrated in Hadoop-Hdfs-trunk-Commit #2774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2774/)
HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

Uploading the branch-1 version (same change).

I've committed this and merged to branch-2, branch-2.0.2-alpha, and branch-1. Thanks JD!

Integrated in Hadoop-Common-trunk-Commit #2711 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2711/)
HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

Integrated in Hadoop-Mapreduce-trunk-Commit #2735 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2735/)
HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

Integrated in Hadoop-Hdfs-trunk #1160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1160/)
HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

Integrated in Hadoop-Mapreduce-trunk #1191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1191/)
HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java

Pulled into branch-0.23 (23.6).

Integrated in Hadoop-Hdfs-0.23-Build #491 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/491/)
merge -r 1382408:1382409 for HDFS-2757 (Revision 1431506)

Result = FAILURE
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431506
Files :

• /hadoop/common/branches/branch-0.23
• /hadoop/common/branches/branch-0.23/hadoop-common-project
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
• /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
• /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
• /hadoop/common/branches/branch-0.23/hadoop-project
• /hadoop/common/branches/branch-0.23/hadoop-project/src/site
• /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
• /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt