Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5804

TestMRJobsWithProfiler#testProfiler timesout

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      testProfiler(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)  Time elapsed: 154.972 sec  <<< ERROR!
      java.lang.Exception: test timed out after 120000 milliseconds
      	at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
      	at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
      	at java.io.File.exists(File.java:813)
      	at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
      	at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
      	at org.apache.log4j.Category.forcedLog(Category.java:391)
      	at org.apache.log4j.Category.log(Category.java:856)
      	at org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:208)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
      	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1570)
      	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:311)
      	at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:599)
      	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1344)
      	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1306)
      	at org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfiler(TestMRJobsWithProfiler.java:138)
      
      
      Results :
      
      Tests in error: 
        TestMRJobsWithProfiler.testProfiler:138 »  test timed out after 120000 millise...
      
      1. LOG.txt
        115 kB
        Mit Desai
      2. MAPREDUCE-5804.patch
        0.9 kB
        Mit Desai

        Issue Links

          Activity

          Hide
          Gera Shegalov added a comment -

          Myt Dezai, Is there some statistics showing that this test is flapping? It does not show up in our builds.

          Show
          Gera Shegalov added a comment - Myt Dezai , Is there some statistics showing that this test is flapping? It does not show up in our builds.
          Hide
          Mit Desai added a comment -

          I have attached the log.txt that has the standard output from our nightly build. I have tried reproducing the failure but I was only able to do that once in many times. And lost the surefire-report as I was running the test in a loop.

          I think the timeout is due to the machine load. I have a patch that I will be uploading soon. It increases the timeout value so the test has enough time even when tons of jobs are running.

          Show
          Mit Desai added a comment - I have attached the log.txt that has the standard output from our nightly build. I have tried reproducing the failure but I was only able to do that once in many times. And lost the surefire-report as I was running the test in a loop. I think the timeout is due to the machine load. I have a patch that I will be uploading soon. It increases the timeout value so the test has enough time even when tons of jobs are running.
          Hide
          Mit Desai added a comment -

          Attaching the patch for trunk and branch-2

          Show
          Mit Desai added a comment - Attaching the patch for trunk and branch-2
          Hide
          Akira AJISAKA added a comment -

          The timeout occurred in Jenkins and I filed MAPREDUCE-5793. https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/org.apache.hadoop.mapreduce.v2/TestMRJobsWithProfiler/testProfiler
          I thought it would be better to get the test faster instead of simply increasing the timeout, however, I'm okay to increase the timeout because now I suppose it's difficult to make it faster.
          I'll close MAPREDUCE-5793.

          Show
          Akira AJISAKA added a comment - The timeout occurred in Jenkins and I filed MAPREDUCE-5793 . https://builds.apache.org/job/PreCommit-HADOOP-Build/3656//testReport/org.apache.hadoop.mapreduce.v2/TestMRJobsWithProfiler/testProfiler I thought it would be better to get the test faster instead of simply increasing the timeout, however, I'm okay to increase the timeout because now I suppose it's difficult to make it faster. I'll close MAPREDUCE-5793 .
          Hide
          Mit Desai added a comment -

          Thanks Akira AJISAKA. I did not find MAPREDUCE-5793. Or else I would have posted the comments there.

          Show
          Mit Desai added a comment - Thanks Akira AJISAKA . I did not find MAPREDUCE-5793 . Or else I would have posted the comments there.
          Hide
          Chen He added a comment -

          +1, download the patch, apply to trunk, run the test, and no timeout reported.

          Show
          Chen He added a comment - +1, download the patch, apply to trunk, run the test, and no timeout reported.
          Hide
          Kihwal Lee added a comment -

          +1

          Show
          Kihwal Lee added a comment - +1
          Hide
          Kihwal Lee added a comment -

          Thanks for working on this. I've committed this to trunk and branch-2.

          Show
          Kihwal Lee added a comment - Thanks for working on this. I've committed this to trunk and branch-2.
          Hide
          Mit Desai added a comment -

          Thanks Kihwal Lee

          Show
          Mit Desai added a comment - Thanks Kihwal Lee
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5371 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5371/)
          MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5371 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5371/ ) MAPREDUCE-5804 . TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #517 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/517/)
          MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #517 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/517/ ) MAPREDUCE-5804 . TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1709 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1709/)
          MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1709 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1709/ ) MAPREDUCE-5804 . TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1734 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1734/)
          MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1734 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1734/ ) MAPREDUCE-5804 . TestMRJobsWithProfiler#testProfiler timesout. Contributed by Mit Desai. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1579995 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithProfiler.java
          Hide
          sam liu added a comment -

          I applied this fix, but still encounter timeout issue. After changing '150000' to '250000', this test could pass on my env. So could we enlarge the timeout value, like to '250000'?

          Show
          sam liu added a comment - I applied this fix, but still encounter timeout issue. After changing '150000' to '250000', this test could pass on my env. So could we enlarge the timeout value, like to '250000'?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Closing old tickets that are already part of a release.

          Show
          Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already part of a release.

            People

            • Assignee:
              Mit Desai
              Reporter:
              Mit Desai
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development