Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Observed in QA report of YARN-3840

      Running org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
      Tests run: 9, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 20.699 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
      testMRAppMasterMidLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)  Time elapsed: 0.474 sec  <<< FAILURE!
      java.lang.AssertionError: null
      	at org.junit.Assert.fail(Assert.java:86)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.junit.Assert.assertTrue(Assert.java:52)
      	at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock(TestMRAppMaster.java:174)
      
      testMRAppMasterSuccessLock(org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster)  Time elapsed: 0.175 sec  <<< ERROR!
      org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File file:/home/varun/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004-1448100479292-TestAppMasterUser-%3Cmissing+job+name%3E-1448100479413-0-0-SUCCEEDED-default-1448100479292.jhist_tmp does not exist
      	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
      	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
      	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
      	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
      	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1154)
      	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
      	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
      	at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
      	at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
      	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
      	at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
      

        Issue Links

          Activity

          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The log can be seen here: https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2649/testReport/

          About org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock:

          java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist
          
          Stack trace
          org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist
          	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
          	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
          	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
          	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
          	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
          	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
          	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
          	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
          	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1153)
          	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
          	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
          	at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
          	at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
          	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
          	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
          	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
          	at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
          

          About org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock:

          java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist
          
          Stack trace
          org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist
          	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
          	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
          	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
          	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
          	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
          	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372)
          	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513)
          	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346)
          	at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1153)
          	at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
          	at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
          	at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
          	at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
          	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751)
          	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
          	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247)
          	at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - The log can be seen here: https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2649/testReport/ About org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist Stack trace org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1153) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247) at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254) About org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterMidLock: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist Stack trace org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.FileNotFoundException: File file:/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/staging/history/done_intermediate/TestAppMasterUser/job_1317529182569_0004_conf.xml_tmp does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:372) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:513) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.moveTmpToDone(JobHistoryEventHandler.java:1346) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processDoneFiles(JobHistoryEventHandler.java:1153) at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157) at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1751) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1247) at org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster.testMRAppMasterSuccessLock(TestMRAppMaster.java:254)
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I ran git bisect and detected that MAPREDUCE-5485 broke the test.

          Show
          ajisakaa Akira Ajisaka added a comment - I ran git bisect and detected that MAPREDUCE-5485 broke the test.
          Hide
          djp Junping Du added a comment -

          Upload a quick fix to test failure.

          Show
          djp Junping Du added a comment - Upload a quick fix to test failure.
          Hide
          djp Junping Du added a comment -

          The previous failure is because since MAPREDUCE-5485, we allow MR job can retry on AM failure during committing stage (if Committer is repeatable). So MRAppMaster.initAndStartAppMaster() won't throw fatal exception if there are commit start file exists (which hints previous AM failed in the middle of commit) for FileOutputCommitter which is default for version 2 algorithm in trunk. I think we don't need this fix in branch-2 as the version in branch-2 is 1.

          Show
          djp Junping Du added a comment - The previous failure is because since MAPREDUCE-5485 , we allow MR job can retry on AM failure during committing stage (if Committer is repeatable). So MRAppMaster.initAndStartAppMaster() won't throw fatal exception if there are commit start file exists (which hints previous AM failed in the middle of commit) for FileOutputCommitter which is default for version 2 algorithm in trunk. I think we don't need this fix in branch-2 as the version in branch-2 is 1.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 46s trunk passed
          +1 compile 0m 19s trunk passed with JDK v1.8.0_66
          +1 compile 0m 22s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 11s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 44s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 18s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.8.0_66
          +1 javac 0m 20s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.7.0_85
          +1 javac 0m 22s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 28s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 0m 15s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 18s the patch passed with JDK v1.7.0_85
          +1 unit 9m 3s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66.
          +1 unit 9m 45s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_85.
          -1 asflicense 0m 24s Patch generated 7 ASF License warnings.
          34m 18s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774057/MAPREDUCE-6555.patch
          JIRA Issue MAPREDUCE-6555
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4a114571b146 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28dfe72
          findbugs v3.0.0
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 46s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_66 +1 compile 0m 22s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 18s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 25s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_66 +1 javac 0m 20s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_85 +1 javac 0m 22s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 51s the patch passed +1 javadoc 0m 15s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 18s the patch passed with JDK v1.7.0_85 +1 unit 9m 3s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_66. +1 unit 9m 45s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 24s Patch generated 7 ASF License warnings. 34m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774057/MAPREDUCE-6555.patch JIRA Issue MAPREDUCE-6555 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4a114571b146 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28dfe72 findbugs v3.0.0 JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6185/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          LGTM. Thanks Junping Du for the quick response.
          Now I cannot reproduce the failure of testMRAppMasterSuccessLock. Hi Varun Saxena and Tsuyoshi Ozawa, is the failure still a problem?

          Show
          ajisakaa Akira Ajisaka added a comment - LGTM. Thanks Junping Du for the quick response. Now I cannot reproduce the failure of testMRAppMasterSuccessLock . Hi Varun Saxena and Tsuyoshi Ozawa , is the failure still a problem?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          -1 asflicense 0m 24s Patch generated 7 ASF License warnings.

          TestMRAppMaster is using directory other than test.build.data, so the intermediate files are checked by Apache Rat. We should fix it in a separate jira.

          Show
          ajisakaa Akira Ajisaka added a comment - -1 asflicense 0m 24s Patch generated 7 ASF License warnings. TestMRAppMaster is using directory other than test.build.data , so the intermediate files are checked by Apache Rat. We should fix it in a separate jira.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I'm checking by running the test case. Please wait a moment.

          Show
          ozawa Tsuyoshi Ozawa added a comment - I'm checking by running the test case. Please wait a moment.
          Hide
          varun_saxena Varun Saxena added a comment -

          +1. Test passes for me.

          Show
          varun_saxena Varun Saxena added a comment - +1. Test passes for me.
          Hide
          djp Junping Du added a comment -

          TestMRAppMaster is using directory other than test.build.data, so the intermediate files are checked by Apache Rat. We should fix it in a separate jira.

          Agree. The worse thing is the test doesn't cleanup the directory in the end because the cleanup() get called before every tests rather than after. Let's file a separate JIRA to fix them.

          Show
          djp Junping Du added a comment - TestMRAppMaster is using directory other than test.build.data, so the intermediate files are checked by Apache Rat. We should fix it in a separate jira. Agree. The worse thing is the test doesn't cleanup the directory in the end because the cleanup() get called before every tests rather than after. Let's file a separate JIRA to fix them.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          We should fix it in a separate jira.

          This issue is tracked by MAPREDUCE-6557.

          Show
          ajisakaa Akira Ajisaka added a comment - We should fix it in a separate jira. This issue is tracked by MAPREDUCE-6557 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Junping Du for the comment. Updated the patch to cleanup the test dir after the tests.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Junping Du for the comment. Updated the patch to cleanup the test dir after the tests.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          My mistake. I attached the patch to the wrong jira. Deleting the patch.

          Show
          ajisakaa Akira Ajisaka added a comment - My mistake. I attached the patch to the wrong jira. Deleting the patch.
          Hide
          djp Junping Du added a comment -

          I'm checking by running the test case. Please wait a moment.

          The Jenkins test report above already show it. Isn't it?

          Show
          djp Junping Du added a comment - I'm checking by running the test case. Please wait a moment. The Jenkins test report above already show it. Isn't it?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 57s trunk passed
          +1 compile 0m 23s trunk passed with JDK v1.8.0_66
          +1 compile 0m 25s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 31s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 49s trunk passed
          +1 javadoc 0m 18s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 19s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 28s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.8.0_66
          +1 javac 0m 22s the patch passed
          +1 compile 0m 25s the patch passed with JDK v1.7.0_85
          +1 javac 0m 25s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 59s the patch passed
          +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 19s the patch passed with JDK v1.7.0_85
          -1 unit 10m 0s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66.
          -1 unit 10m 20s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          37m 55s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.TestMRAppMaster
          JDK v1.7.0_85 Failed junit tests hadoop.mapreduce.v2.app.TestMRAppMaster



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774341/MAPREDUCE-6555.02.patch
          JIRA Issue MAPREDUCE-6555
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e8afc7dd9088 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 177975e
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 57s trunk passed +1 compile 0m 23s trunk passed with JDK v1.8.0_66 +1 compile 0m 25s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 18s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 19s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 28s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_66 +1 javac 0m 22s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_85 +1 javac 0m 25s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 59s the patch passed +1 javadoc 0m 19s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 19s the patch passed with JDK v1.7.0_85 -1 unit 10m 0s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_66. -1 unit 10m 20s hadoop-mapreduce-client-app in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 37m 55s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.app.TestMRAppMaster JDK v1.7.0_85 Failed junit tests hadoop.mapreduce.v2.app.TestMRAppMaster Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12774341/MAPREDUCE-6555.02.patch JIRA Issue MAPREDUCE-6555 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e8afc7dd9088 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 177975e findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6194/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Junping Du yeah. However, we have lots intermittent test failures, so I've checked it. +1.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Junping Du yeah. However, we have lots intermittent test failures, so I've checked it. +1.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1

          Show
          ajisakaa Akira Ajisaka added a comment - +1
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Jenkins CI looks to be checking another patch. +1 for MAPREDUCE-6555.patch. Checking this in.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Jenkins CI looks to be checking another patch. +1 for MAPREDUCE-6555 .patch. Checking this in.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk. Thanks Junping Du for the contribution and Akira Ajisaka and Varun Saxena for the review and comments.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk. Thanks Junping Du for the contribution and Akira Ajisaka and Varun Saxena for the review and comments.
          Hide
          djp Junping Du added a comment -

          Thanks Varun Saxena for reporting it and Tsuyoshi Ozawa and Akira Ajisaka for review!

          Show
          djp Junping Du added a comment - Thanks Varun Saxena for reporting it and Tsuyoshi Ozawa and Akira Ajisaka for review!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8886 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8886/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8886 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8886/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2664 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2664/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2664 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2664/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #733 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/733/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #733 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/733/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1454 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1454/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1454 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1454/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/722/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/722/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/643/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #643 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/643/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2580 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2580/)
          MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2580 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2580/ ) MAPREDUCE-6555 . TestMRAppMaster fails on trunk. (Junping Du via ozawa) (ozawa: rev 3e85542a7afb8bbbc19f2aa59da04ec824168c0e) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java hadoop-mapreduce-project/CHANGES.txt

            People

            • Assignee:
              djp Junping Du
              Reporter:
              varun_saxena Varun Saxena
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development