Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1950

Blocks that are under construction are not getting read if the blocks are more than 10. Only complete blocks are read properly.

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Target Version/s:

      Description

      Before going to the root cause lets see the read behavior for a file having more than 10 blocks in append case..
      Logic:
      ====
      There is prefetch size dfs.read.prefetch.size for the DFSInputStream which has default value of 10
      This prefetch size is the number of blocks that the client will fetch from the namenode for reading a file..
      For example lets assume that a file X having 22 blocks is residing in HDFS
      The reader first fetches first 10 blocks from the namenode and start reading
      After the above step , the reader fetches the next 10 blocks from NN and continue reading
      Then the reader fetches the remaining 2 blocks from NN and complete the write
      Cause:
      =======
      Lets see the cause for this issue now...
      Scenario that will fail is "Writer wrote 10+ blocks and a partial block and called sync. Reader trying to read the file will not get the last partial block" .

      Client first gets the 10 block locations from the NN. Now it checks whether the file is under construction and if so it gets the size of the last partial block from datanode and reads the full file
      However when the number of blocks is more than 10, the last block will not be in the first fetch. It will be in the second or other blocks(last block will be in (num of blocks / 10)th fetch)
      The problem now is, in DFSClient there is no logic to get the size of the last partial block(as in case of point 1), for the rest of the fetches other than first fetch, the reader will not be able to read the complete data synced...........!!

      also the InputStream.available api uses the first fetched block size to iterate. Ideally this size has to be increased

      1. HDFS-1950.1.patch
        16 kB
        Uma Maheswara Rao G
      2. hdfs-1950-0.20-append-tests.txt
        6 kB
        Todd Lipcon
      3. HDFS-1950-2.patch
        20 kB
        ramkrishna.s.vasudevan
      4. hdfs-1950-trunk-test.txt
        4 kB
        Todd Lipcon
      5. hdfs-1950-trunk-test.txt
        4 kB
        Todd Lipcon

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480115/HDFS-1950-2.patch
        against trunk revision 1126312.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/612//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480115/HDFS-1950-2.patch against trunk revision 1126312. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/612//console This message is automatically generated.
        Hide
        ramkrishna.s.vasudevan added a comment -

        This patch applies for 0.20 append branch.

        Show
        ramkrishna.s.vasudevan added a comment - This patch applies for 0.20 append branch.
        Hide
        Todd Lipcon added a comment -

        Cleaned up and updated the test case for trunk to make sure this bug does not exist there. Luckily it passes I'd like to commit these tests to trunk anyway - good coverage.

        I also verified the failures on the append branch, so will review the code changes there.

        Show
        Todd Lipcon added a comment - Cleaned up and updated the test case for trunk to make sure this bug does not exist there. Luckily it passes I'd like to commit these tests to trunk anyway - good coverage. I also verified the failures on the append branch, so will review the code changes there.
        Hide
        Todd Lipcon added a comment -

        same patch with shorter test method names

        Show
        Todd Lipcon added a comment - same patch with shorter test method names
        Hide
        Todd Lipcon added a comment -

        Hi Ramkrishna. Could you please explain in some detail the changes in FSNamesystem? It seems like very messy code – eg passing a Boolean[] around in order to add "return arguments", etc? Can this be done more cleanly? Having code like that in FSN scares me.

        Show
        Todd Lipcon added a comment - Hi Ramkrishna. Could you please explain in some detail the changes in FSNamesystem? It seems like very messy code – eg passing a Boolean[] around in order to add "return arguments", etc? Can this be done more cleanly? Having code like that in FSN scares me.
        Hide
        Todd Lipcon added a comment -

        cleaned up tests on 0.20-append

        Show
        Todd Lipcon added a comment - cleaned up tests on 0.20-append
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481633/hdfs-1950-0.20-append-tests.txt
        against trunk revision 1132779.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/731//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481633/hdfs-1950-0.20-append-tests.txt against trunk revision 1132779. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/731//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481631/hdfs-1950-trunk-test.txt
        against trunk revision 1132779.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481631/hdfs-1950-trunk-test.txt against trunk revision 1132779. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/729//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12481632/hdfs-1950-trunk-test.txt
        against trunk revision 1132779.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestHDFSCLI

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481632/hdfs-1950-trunk-test.txt against trunk revision 1132779. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/730//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        In my cluster also, i have seen this problem.
        Changing it to blocker.

        Show
        Uma Maheswara Rao G added a comment - In my cluster also, i have seen this problem. Changing it to blocker.
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Todd,

        I have updated the patch for review!

        HDFS-1950.1.patch: I merged your re-factored tests as well.

        Can you please have a look on it?

        Show
        Uma Maheswara Rao G added a comment - Hi Todd, I have updated the patch for review! HDFS-1950 .1.patch: I merged your re-factored tests as well. Can you please have a look on it?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12490392/HDFS-1950.1.patch
        against trunk revision 1157232.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1103//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12490392/HDFS-1950.1.patch against trunk revision 1157232. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1103//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        This patch is for 20Append!

        Show
        Uma Maheswara Rao G added a comment - This patch is for 20Append!
        Hide
        Uma Maheswara Rao G added a comment -

        Patch (HDFS-1950.1.patch) ready for review!

        Show
        Uma Maheswara Rao G added a comment - Patch ( HDFS-1950 .1.patch) ready for review!
        Hide
        Uma Maheswara Rao G added a comment -

        Hi Todd,
        Can you please take a look on this?

        --thanks

        Show
        Uma Maheswara Rao G added a comment - Hi Todd, Can you please take a look on this? --thanks
        Hide
        Uma Maheswara Rao G added a comment -

        Hi,

        Fix is basically for two scenarios.

        1) DFSClient side changes is basically to ensure reading the partial block.

        2) Problem here is, DFClient will go for next fetch based on the blocks size.

        Consider a corner boundary case (take prefetch size as 10 *blocksize), number of blocks are exactly same as 10.5 or 20.5 ...etc can create problem, because clinet will not even bother for next fetch because initially he will not know the size of that partial block. He will know only that 10blocks size.

        To fix this problem, we have introduce on check in FSNameSystem.

             ......
             ......
             LocatedBlocks createLocatedBlocks = inode.createLocatedBlocks(results);
        
             createLocatedBlocksForThePartialBlock(inode, blocks, curPos,
        			createLocatedBlocks);
        
             return createLocatedBlocks;
          }
        
        
           private void createLocatedBlocksForThePartialBlock(INodeFile inode,
        		Block[] blocks, long curPos, LocatedBlocks createLocatedBlocks) {
        	int curBlk;
        	if(blocks.length > PRE_FETCH_SIZE &&
        			blocks.length % PRE_FETCH_SIZE == 1 && createLocatedBlocks.getFileLength() == curPos)
            {
        
           .........
           ........
        

        When blocks are exactly 10.5, then FSNameSystem will populate the 0.5th block id also. So, that client anyway will update the partial block size. Client can take care of reading this boundary partial block.

        This patch is basically for review. Here one more problem i wanted to raise is that, prefetch size is configured.
        In this patch i put that value as 10.

         prefetchSize = conf.getLong("dfs.read.prefetch.size", prefetchSize);
         

        I am planning to include this property in server namenode as well, because of above reason. or Do yu have any other suggestion.

        Once the patch is approved i will prepare it for 20Append and 205 branches with the provided suggestions.

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Hi, Fix is basically for two scenarios. 1) DFSClient side changes is basically to ensure reading the partial block. 2) Problem here is, DFClient will go for next fetch based on the blocks size. Consider a corner boundary case (take prefetch size as 10 *blocksize), number of blocks are exactly same as 10.5 or 20.5 ...etc can create problem, because clinet will not even bother for next fetch because initially he will not know the size of that partial block. He will know only that 10blocks size. To fix this problem, we have introduce on check in FSNameSystem. ...... ...... LocatedBlocks createLocatedBlocks = inode.createLocatedBlocks(results); createLocatedBlocksForThePartialBlock(inode, blocks, curPos, createLocatedBlocks); return createLocatedBlocks; } private void createLocatedBlocksForThePartialBlock(INodeFile inode, Block[] blocks, long curPos, LocatedBlocks createLocatedBlocks) { int curBlk; if (blocks.length > PRE_FETCH_SIZE && blocks.length % PRE_FETCH_SIZE == 1 && createLocatedBlocks.getFileLength() == curPos) { ......... ........ When blocks are exactly 10.5, then FSNameSystem will populate the 0.5th block id also. So, that client anyway will update the partial block size. Client can take care of reading this boundary partial block. This patch is basically for review. Here one more problem i wanted to raise is that, prefetch size is configured. In this patch i put that value as 10. prefetchSize = conf.getLong( "dfs.read.prefetch.size" , prefetchSize); I am planning to include this property in server namenode as well, because of above reason. or Do yu have any other suggestion. Once the patch is approved i will prepare it for 20Append and 205 branches with the provided suggestions. Thanks Uma
        Hide
        Steve Loughran added a comment -

        Would be nice if the config parameter wasn't another string embedded in the src. We have too many of those. In an ideal world there'd be an Interface "DfsConfigOptions" with every string listed and javadoc'd. Even if it doesn't exist over everything, there's no reason to continue existing bad habits.

        Show
        Steve Loughran added a comment - Would be nice if the config parameter wasn't another string embedded in the src. We have too many of those. In an ideal world there'd be an Interface "DfsConfigOptions" with every string listed and javadoc'd. Even if it doesn't exist over everything, there's no reason to continue existing bad habits.
        Hide
        Uma Maheswara Rao G added a comment -

        Thanks Steve for taking a look.

        This confuguration not yet included in patch. Once config parameter is finalized for NN side, i will include it in DFSConfigKeys.

        @Jitendra,
        Can you please take a look? testWhenSizeHasPartialBlockwithMultiplesOf10 will give you the answer for change in FSNameSystem.java. But here it required prefetch size in FSNameSystem as well. That is presently client side configuration. Do you have any suggestion for handling this conguration at server side?

        Thanks
        Uma

        Show
        Uma Maheswara Rao G added a comment - Thanks Steve for taking a look. This confuguration not yet included in patch. Once config parameter is finalized for NN side, i will include it in DFSConfigKeys. @Jitendra, Can you please take a look? testWhenSizeHasPartialBlockwithMultiplesOf10 will give you the answer for change in FSNameSystem.java. But here it required prefetch size in FSNameSystem as well. That is presently client side configuration. Do you have any suggestion for handling this conguration at server side? Thanks Uma
        Hide
        Matt Foley added a comment -

        moved target version to 1.2.0 upon publishing 1.1.1 RC.

        Show
        Matt Foley added a comment - moved target version to 1.2.0 upon publishing 1.1.1 RC.
        Hide
        Matt Foley added a comment -

        Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

        Show
        Matt Foley added a comment - Changed Target Version to 1.3.0 upon release of 1.2.0. Please change to 1.2.1 if you intend to submit a fix for branch-1.2.

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:

              Development