Suhas Gogate, can you please add details about what test you are doing and what issues you are seeing?

In function getBlockLocalPathInfo the input parameter block is passed by the client. Since client will buffer file's metadata, block.getGenerationStamp() may be older then the real generationStamp on Datanode. Datanode will report that it cannot find metadata file and then client fail to read.

@Override // FsDatasetSpi
  public BlockLocalPathInfo getBlockLocalPathInfo(ExtendedBlock block)
      throws IOException {
    File datafile = getBlockFile(block);
    File metafile = FsDatasetUtil.getMetaFile(datafile, block.getGenerationStamp());
    BlockLocalPathInfo info = new BlockLocalPathInfo(block,
        datafile.getAbsolutePath(), metafile.getAbsolutePath());
    return info;
  }

Test case

enable read-circuit and set dfs.client.use.legacy.blockreader.local to true
1) crete a file with two blocks.
2) open it for read, but not read. (client fetch block metadata)
3) append to it. (increase generation stamp of last block)
4) continue to read. (will fail)

I see. The problem is only in legacy.blockreader.local, which uses the generation stamp passed by client. Just have checked the remote block reader. It does not have such bug.

h7885_20150305.patch: get replica from volume map.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702821/h7885_20150305.patch
against trunk revision 1b67209.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead

The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9751//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9751//console

This message is automatically generated.

Thanks for working on this, Nicholas! Looks like the failure was caused by the change on the test TestShortCircuitLocalRead:

java.io.EOFException: End of file reached before reading fully.
       at org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.checkFileContent(TestShortCircuitLocalRead.java:167)
     at org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.doTestShortCircuitReadImpl(TestShortCircuitLocalRead.java:287)
 at org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.doTestShortCircuitRead(TestShortCircuitLocalRead.java:244)
 at org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead.testReadFromAnOffset(TestShortCircuitLocalRead.java:333)

I.e., the reader could not read the newly appended data.

Besides, I guess we only need to add extra tests for legacy local reader?

Jing, thanks for the comments.

h7885_20150306.patch: add a new test in TestBlockReaderLocalLegacy

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702968/h7885_20150306.patch
against trunk revision 952640f.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9763//console

This message is automatically generated.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702968/h7885_20150306.patch
against trunk revision 952640f.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9764//console

This message is automatically generated.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702968/h7885_20150306.patch
against trunk revision 952640f.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9765//console

This message is automatically generated.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702968/h7885_20150306.patch
against trunk revision 952640f.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9766//console

This message is automatically generated.

> -1 patch. Trunk compilation may be broken.

Neither trunk nor the patch has a compilation problem. It was the machine.

• https://builds.apache.org/job/PreCommit-HDFS-Build/9763/artifact/patchprocess/trunkJavacWarnings.txt
  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean) on project hadoop-hdfs: Failed to clean project: Failed to delete /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3 -> [Help 1]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12702968/h7885_20150306.patch
against trunk revision 95bfd08.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.namenode.TestFileTruncate

The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9775//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9775//console

This message is automatically generated.

The latest patch looks good to me. The failed tests should be unrelated. +1.

I will commit it shortly.

Thanks for the fix, Nicholas! I've committed this to trunk and branch-2.

SUCCESS: Integrated in Hadoop-trunk-Commit #7271 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7271/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/125/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Yarn-trunk #859 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/859/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2057/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/116/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #125 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/125/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2075 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2075/)
HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. (jing9: rev 24db0812be64e83a48ade01fc1eaaeaedad4dec0)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
• hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

Sangjin Lee backported this to 2.6.1, the patch applies cleanly. I just pushed the commit to 2.6.1 after running compilation and TestBlockReaderLocalLegacy which changed in the patch.