Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2757

Cannot read a local block that's being written to when using the local read short circuit

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0, 2.0.2-alpha, 0.23.6
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When testing the tail'ing of a local file with the read short circuit on, I get:

      2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal requested with incorrect offset:  Offset 0 and length 8230400 don't match block blk_-2842916025951313698_454072 ( blockLen 124 )
      2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_-2842916025951313698_454072 from cache because local file /export4/jdcryans/dfs/data/blocksBeingWritten/blk_-2842916025951313698 could not be opened.
      2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_-2842916025951313698_454072 on local machine java.io.IOException:  Offset 0 and length 8230400 don't match block blk_-2842916025951313698_454072 ( blockLen 124 )
      2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Try reading via the datanode on /10.4.13.38:51010
      java.io.EOFException: hdfs://sv4r11s38:9100/hbase-1/.logs/sv4r13s38,62023,1325808100311/sv4r13s38%2C62023%2C1325808100311.1325808100818, entryStart=7190409, pos=8230400, end=8230400, edit=5
      
      1. HDFS-2757-branch-1.patch
        1 kB
        Jean-Daniel Cryans
      2. HDFS-2757-branch-1-v2.patch
        0.7 kB
        Jean-Daniel Cryans
      3. HDFS-2757-trunk.patch
        2 kB
        Jean-Daniel Cryans
      4. hdfs-2757.txt
        2 kB
        Eli Collins
      5. hdfs-2757-b1.txt
        1 kB
        Eli Collins

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #491 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/491/)
        merge -r 1382408:1382409 for HDFS-2757 (Revision 1431506)

        Result = FAILURE
        kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431506
        Files :

        • /hadoop/common/branches/branch-0.23
        • /hadoop/common/branches/branch-0.23/hadoop-common-project
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
        • /hadoop/common/branches/branch-0.23/hadoop-project
        • /hadoop/common/branches/branch-0.23/hadoop-project/src/site
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #491 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/491/ ) merge -r 1382408:1382409 for HDFS-2757 (Revision 1431506) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431506 Files : /hadoop/common/branches/branch-0.23 /hadoop/common/branches/branch-0.23/hadoop-common-project /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples /hadoop/common/branches/branch-0.23/hadoop-project /hadoop/common/branches/branch-0.23/hadoop-project/src/site /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
        Hide
        Kihwal Lee added a comment -

        Pulled into branch-0.23 (23.6).

        Show
        Kihwal Lee added a comment - Pulled into branch-0.23 (23.6).
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1191/)
        HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1191/ ) HDFS-2757 . Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1160/)
        HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1160/ ) HDFS-2757 . Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2735 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2735/)
        HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2735 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2735/ ) HDFS-2757 . Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2711 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2711/)
        HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2711 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2711/ ) HDFS-2757 . Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2, branch-2.0.2-alpha, and branch-1. Thanks JD!

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2, branch-2.0.2-alpha, and branch-1. Thanks JD!
        Hide
        Eli Collins added a comment -

        Uploading the branch-1 version (same change).

        Show
        Eli Collins added a comment - Uploading the branch-1 version (same change).
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2774/)
        HDFS-2757. Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2774/ ) HDFS-2757 . Cannot read a local block that's being written to when using the local read short circuit. Contributed by Jean-Daniel Cryans (Revision 1382409) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382409 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        Hide
        stack added a comment -

        +1 on Eli patch

        Show
        stack added a comment - +1 on Eli patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12544365/hdfs-2757.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544365/hdfs-2757.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3163//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        I thought both findbugs warnings were bogus but actually the 2nd one is due to this patch. The access to locatedBlocks needs to be synchronized, in practice it should be fine, but findbugs complains because all other accesses are synchronized. I pulled your check out to a private synchronized method. Updated patch attached.

        Show
        Eli Collins added a comment - I thought both findbugs warnings were bogus but actually the 2nd one is due to this patch. The access to locatedBlocks needs to be synchronized, in practice it should be fine, but findbugs complains because all other accesses are synchronized. I pulled your check out to a private synchronized method. Updated patch attached.
        Hide
        Eli Collins added a comment -

        +1 looks good, thanks JD!

        Show
        Eli Collins added a comment - +1 looks good, thanks JD!
        Hide
        Jean-Daniel Cryans added a comment -

        TestHftpDelegationToken fails even without my patch.

        Show
        Jean-Daniel Cryans added a comment - TestHftpDelegationToken fails even without my patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12542461/HDFS-2757-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestHftpDelegationToken

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542461/HDFS-2757-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHftpDelegationToken +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3099//console This message is automatically generated.
        Hide
        Jean-Daniel Cryans added a comment -

        Attaching trunk's version. It's been a while since I've been in trunk so apologies if I did anything wrong.

        It seems the input stream was moved out of DFSClient so I had to do some massaging. Basically I'm trying to keep shouldTryShortCircuitRead the same way I changed it for branch-1 but now the check is done in DFSInputStream.

        Show
        Jean-Daniel Cryans added a comment - Attaching trunk's version. It's been a while since I've been in trunk so apologies if I did anything wrong. It seems the input stream was moved out of DFSClient so I had to do some massaging. Basically I'm trying to keep shouldTryShortCircuitRead the same way I changed it for branch-1 but now the check is done in DFSInputStream.
        Hide
        stack added a comment -

        You have patch for trunk J-D so can run it by hadoopqa?

        Show
        stack added a comment - You have patch for trunk J-D so can run it by hadoopqa?
        Hide
        Jean-Daniel Cryans added a comment -

        We've been running with my patch in production for 2 weeks now on a few hundred nodes with HBase doing reads on local files that are still being written to. Everything looks good.

        Show
        Jean-Daniel Cryans added a comment - We've been running with my patch in production for 2 weeks now on a few hundred nodes with HBase doing reads on local files that are still being written to. Everything looks good.
        Hide
        Jean-Daniel Cryans added a comment -

        I guess I was tired when I did the other patch and missed the refactoring opportunity, attaching new version against branch-1.

        Show
        Jean-Daniel Cryans added a comment - I guess I was tired when I did the other patch and missed the refactoring opportunity, attaching new version against branch-1.
        Hide
        Jean-Daniel Cryans added a comment -

        I still don't understand what's the length mismatch here, but one way we can fix this is by not using the BlockReaderLocal when the block is being written to. This is a 2 lines change which I'm attaching here. I tested it on a single node and it works, I'm about to deploy on a few clusters.

        Show
        Jean-Daniel Cryans added a comment - I still don't understand what's the length mismatch here, but one way we can fix this is by not using the BlockReaderLocal when the block is being written to. This is a 2 lines change which I'm attaching here. I tested it on a single node and it works, I'm about to deploy on a few clusters.
        Hide
        Jean-Daniel Cryans added a comment -

        Another failure mode is when the block is done being written to (block unrelated with the one in the description):

        2012-08-10 00:17:02,804 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_5190712822654892045_134607537 from cache because local file /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 could not be opened.
        2012-08-10 00:17:02,804 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_5190712822654892045_134607537 on local machine java.io.FileNotFoundException: /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 (No such file or directory)
        
        Show
        Jean-Daniel Cryans added a comment - Another failure mode is when the block is done being written to (block unrelated with the one in the description): 2012-08-10 00:17:02,804 WARN org.apache.hadoop.hdfs.DFSClient: BlockReaderLocal: Removing blk_5190712822654892045_134607537 from cache because local file /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 could not be opened. 2012-08-10 00:17:02,804 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read block blk_5190712822654892045_134607537 on local machine java.io.FileNotFoundException: /export2/hadoop/dfs/data/blocksBeingWritten/blk_5190712822654892045 (No such file or directory)
        Hide
        Jean-Daniel Cryans added a comment -

        Suresh, I changed the title to reflect what I think is really the issue, and yes this is a bad bug.

        Show
        Jean-Daniel Cryans added a comment - Suresh, I changed the title to reflect what I think is really the issue, and yes this is a bad bug.
        Hide
        Suresh Srinivas added a comment -

        so is this still a bug or can it be closed as invalid?

        Show
        Suresh Srinivas added a comment - so is this still a bug or can it be closed as invalid?
        Hide
        Jean-Daniel Cryans added a comment -

        Ah turns out I can read the file, just not the block on the local machine (the "Try reading via the datanode on" didn't lie as I originally thought).

        Show
        Jean-Daniel Cryans added a comment - Ah turns out I can read the file, just not the block on the local machine (the "Try reading via the datanode on" didn't lie as I originally thought).

          People

          • Assignee:
            Jean-Daniel Cryans
            Reporter:
            Jean-Daniel Cryans
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development