Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: test
    • Labels:
      None

      Description

      TestFileInputFormat#testLocality is failing intermittently when verifying each file split has two block locations.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        8m 19s 1 Jason Lowe 06/Jul/12 17:30
        Patch Available Patch Available Resolved Resolved
        2d 23h 46m 1 Robert Joseph Evans 09/Jul/12 17:17
        Resolved Resolved Closed Closed
        94d 1h 31m 1 Arun C Murthy 11/Oct/12 18:48
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Fix Version/s 2.1.0-alpha [ 12321442 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1132/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1132/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #309 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/309/)
        svn merge -c 1359252 FIXES: MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359256)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359256
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #309 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/309/ ) svn merge -c 1359252 FIXES: MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359256) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359256 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1099 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1099/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1099 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1099/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2453 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2453/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2453 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2453/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2503 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2503/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2503 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2503/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2435 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2435/)
        MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2435 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2435/ ) MAPREDUCE-4402 . TestFileInputFormat fails intermittently (Jason Lowe via bobby) (Revision 1359252) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359252 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Fix Version/s 2.0.1-alpha [ 12321442 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Resolution Fixed [ 1 ]
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason,

        +1 I put this into trunk, branch-2, and branch-0.23.

        Show
        Robert Joseph Evans added a comment - Thanks Jason, +1 I put this into trunk, branch-2, and branch-0.23.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12535385/MAPREDUCE-4402.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2547//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2547//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535385/MAPREDUCE-4402.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2547//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2547//console This message is automatically generated.
        Hide
        Robert Joseph Evans added a comment -

        The patch is short and straight to the point. +1 assuming Jenkins comes back OK.

        Show
        Robert Joseph Evans added a comment - The patch is short and straight to the point. +1 assuming Jenkins comes back OK.
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 0.23.3, 2.0.1-alpha, 3.0.0 [ 12320060, 12321442, 12320355 ]
        Jason Lowe made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4402.patch [ 12535385 ]
        Hide
        Jason Lowe added a comment -

        Stacktrace from the failing test:

        junit.framework.AssertionFailedError: expected:<2> but was:<1>
        	at junit.framework.Assert.fail(Assert.java:47)
        	at junit.framework.Assert.failNotEquals(Assert.java:283)
        	at junit.framework.Assert.assertEquals(Assert.java:64)
        	at junit.framework.Assert.assertEquals(Assert.java:195)
        	at junit.framework.Assert.assertEquals(Assert.java:201)
        	at org.apache.hadoop.mapred.TestFileInputFormat.testLocality(TestFileInputFormat.java:86)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at junit.framework.TestCase.runTest(TestCase.java:168)
        	at junit.framework.TestCase.runBare(TestCase.java:134)
        	at junit.framework.TestResult$1.protect(TestResult.java:110)
        	at junit.framework.TestResult.runProtected(TestResult.java:128)
        	at junit.framework.TestResult.run(TestResult.java:113)
        	at junit.framework.TestCase.run(TestCase.java:124)
        	at junit.framework.TestSuite.runTest(TestSuite.java:243)
        	at junit.framework.TestSuite.run(TestSuite.java:238)
        	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
        	at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
        	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
        	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        

        There is a race condition in the test where the block locations can be requested before the namenode is aware of all of the replicas for each block. The test should call DFSTestUtil.waitReplication() to ensure all blocks have their full replication completed before querying for split locations.

        Show
        Jason Lowe added a comment - Stacktrace from the failing test: junit.framework.AssertionFailedError: expected:<2> but was:<1> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:195) at junit.framework.Assert.assertEquals(Assert.java:201) at org.apache.hadoop.mapred.TestFileInputFormat.testLocality(TestFileInputFormat.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) There is a race condition in the test where the block locations can be requested before the namenode is aware of all of the replicas for each block. The test should call DFSTestUtil.waitReplication() to ensure all blocks have their full replication completed before querying for split locations.
        Jason Lowe created issue -

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development