Hadoop Common
  1. Hadoop Common
  2. HADOOP-8599

Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.3, 0.23.1, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None

      Description

      When FileSystem.getFileBlockLocations(file,start,len) is called with "start" argument equal to the file size, the response is not empty. There is a test TestGetFileBlockLocations.testGetFileBlockLocations2 which uses randomly generated "start" and "len" arguments when calling FileSystem.getFileBlockLocations and the test fails randomly (when the generated start value equals to the file size).

        Issue Links

          Activity

          Hide
          Andrey Klochkov added a comment -

          The build https://builds.apache.org/job/Hadoop-Common-0.23-Build/312/ failed due to this bug, this is the failure log:

          org.apache.hadoop.fs.TestGetFileBlockLocations.testGetFileBlockLocations2
          
          Failing for the past 1 build (Since #312 )
          Took 0.29 sec.
          
          Stacktrace
          
          junit.framework.AssertionFailedError: null
          	at junit.framework.Assert.fail(Assert.java:47)
          	at junit.framework.Assert.assertTrue(Assert.java:20)
          	at junit.framework.Assert.assertTrue(Assert.java:27)
          	at org.apache.hadoop.fs.TestGetFileBlockLocations.oneTest(TestGetFileBlockLocations.java:90)
          	at org.apache.hadoop.fs.TestGetFileBlockLocations.__CLR3_0_2h1r7rc12wg(TestGetFileBlockLocations.java:136)
          	at org.apache.hadoop.fs.TestGetFileBlockLocations.testGetFileBlockLocations2(TestGetFileBlockLocations.java:131)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at junit.framework.TestCase.runTest(TestCase.java:168)
          	at junit.framework.TestCase.runBare(TestCase.java:134)
          	at junit.framework.TestResult$1.protect(TestResult.java:110)
          	at junit.framework.TestResult.runProtected(TestResult.java:128)
          	at junit.framework.TestResult.run(TestResult.java:113)
          	at junit.framework.TestCase.run(TestCase.java:124)
          	at junit.framework.TestSuite.runTest(TestSuite.java:243)
          	at junit.framework.TestSuite.run(TestSuite.java:238)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
          	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
          	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
          	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
          Standard Output
          
          2012-07-15 09:13:27,734 INFO  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(50)) - Loaded the native-hadoop library
          
          Show
          Andrey Klochkov added a comment - The build https://builds.apache.org/job/Hadoop-Common-0.23-Build/312/ failed due to this bug, this is the failure log: org.apache.hadoop.fs.TestGetFileBlockLocations.testGetFileBlockLocations2 Failing for the past 1 build (Since #312 ) Took 0.29 sec. Stacktrace junit.framework.AssertionFailedError: null at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertTrue(Assert.java:27) at org.apache.hadoop.fs.TestGetFileBlockLocations.oneTest(TestGetFileBlockLocations.java:90) at org.apache.hadoop.fs.TestGetFileBlockLocations.__CLR3_0_2h1r7rc12wg(TestGetFileBlockLocations.java:136) at org.apache.hadoop.fs.TestGetFileBlockLocations.testGetFileBlockLocations2(TestGetFileBlockLocations.java:131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) Standard Output 2012-07-15 09:13:27,734 INFO util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(50)) - Loaded the native -hadoop library
          Hide
          Andrey Klochkov added a comment -

          The patch fixes the condition of going beyond EOF in the FileSystem.getFileBlockLocations by changing

              if (file.getLen() < start) {
                return new BlockLocation[0];
          

          to

              if (file.getLen() <= start) {
                return new BlockLocation[0];
          

          An additional assert is added to the unit tests to verify the case.

          Show
          Andrey Klochkov added a comment - The patch fixes the condition of going beyond EOF in the FileSystem.getFileBlockLocations by changing if (file.getLen() < start) { return new BlockLocation[0]; to if (file.getLen() <= start) { return new BlockLocation[0]; An additional assert is added to the unit tests to verify the case.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536733/HADOOP-8859-branch-0.23.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1198//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1198//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536733/HADOOP-8859-branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1198//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1198//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1, looks good to me. Thanks Andrey.

          Show
          Todd Lipcon added a comment - +1, looks good to me. Thanks Andrey.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2551 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2551/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2551 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2551/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2486/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2486/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2507/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2507/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1106/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1106 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1106/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #316 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/316/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362297)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362297
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #316 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/316/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362297) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362297 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1139/)
          HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1139 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1139/ ) HADOOP-8599 . Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file. Contributed by Andrey Klochkov. (Revision 1362295) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362295 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestGetFileBlockLocations.java
          Hide
          Matt Foley added a comment -

          Fixed Target Versions to be consistent with Fix Versions.
          Removed 1.1.0 target and opened HADOOP-8612 for backport.

          Show
          Matt Foley added a comment - Fixed Target Versions to be consistent with Fix Versions. Removed 1.1.0 target and opened HADOOP-8612 for backport.
          Hide
          Kihwal Lee added a comment -

          I think TestCombineFileInputFormat.testForEmptyFile started failing after this. The split size on an empty input file used to be 1, but it's now 0.

          Show
          Kihwal Lee added a comment - I think TestCombineFileInputFormat.testForEmptyFile started failing after this. The split size on an empty input file used to be 1, but it's now 0.
          Hide
          Kihwal Lee added a comment -

          MAPREDUCE-4470 has been filed.

          Show
          Kihwal Lee added a comment - MAPREDUCE-4470 has been filed.
          Hide
          Mariappan Asokan added a comment -

          Can someone take a look at MAPREDUCE-4470 and propose a proper fix or retract HADOOP-8599 from the trunk? This is causing the trunk build to fail for more than a week. I think returning a 0 split for an empty input is not a good idea. The original behavior of 1 split with size 0 was good. Can others jump in and comment on this?
          Thanks.

          Show
          Mariappan Asokan added a comment - Can someone take a look at MAPREDUCE-4470 and propose a proper fix or retract HADOOP-8599 from the trunk? This is causing the trunk build to fail for more than a week. I think returning a 0 split for an empty input is not a good idea. The original behavior of 1 split with size 0 was good. Can others jump in and comment on this? Thanks.
          Hide
          Shinochika9861 added a comment -

          Thank you for your attention not sending so many junk mails to the mailing list. It's infliction.

          Sent from iPhone.

          Show
          Shinochika9861 added a comment - Thank you for your attention not sending so many junk mails to the mailing list. It's infliction. Sent from iPhone.

            People

            • Assignee:
              Andrey Klochkov
              Reporter:
              Andrey Klochkov
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development