Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 1.0.3
    • Component/s: build
    • Labels:
      None
    • Environment:

      FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)

    • Target Version/s:

      Description

      When running the findbugs target from Jenkins, I get an OutOfMemory error.
      The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.

      We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).

      1. hadoop-7381.patch
        0.8 kB
        Joep Rottinghuis

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop-1.0.3.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop-1.0.3.
          Hide
          Matt Foley added a comment -

          Committed to branch-1 and branch-1.0. Thanks, Joep!

          Show
          Matt Foley added a comment - Committed to branch-1 and branch-1.0. Thanks, Joep!
          Hide
          Matt Foley added a comment -

          Yes, let's get this in the next release.
          +1 for code review.

          Show
          Matt Foley added a comment - Yes, let's get this in the next release. +1 for code review.
          Hide
          Anupam Seth added a comment -

          Seems like we are running into this on branch-1.0. Can someone please commit this so that our builds can be customized to have higher memory for findbugs?

          Show
          Anupam Seth added a comment - Seems like we are running into this on branch-1.0. Can someone please commit this so that our builds can be customized to have higher memory for findbugs?
          Hide
          Joep Rottinghuis added a comment -

          HADOOP-7381 is not really needed for 0.22,
          It was applicable to 0.20-security which is before the project split with a large combined hadoop-common project.
          After the split, hadoop-common is small enough that it does not run OOM with the defaults.
          This does not apply to Mavenization on trunk (at least not in the ant form). I'll have to see if the same exists with Mavenization, but with a different build, the error will be different if it exists at all.

          Show
          Joep Rottinghuis added a comment - HADOOP-7381 is not really needed for 0.22, It was applicable to 0.20-security which is before the project split with a large combined hadoop-common project. After the split, hadoop-common is small enough that it does not run OOM with the defaults. This does not apply to Mavenization on trunk (at least not in the ant form). I'll have to see if the same exists with Mavenization, but with a different build, the error will be different if it exists at all.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12482131/hadoop-7381.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/94//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482131/hadoop-7381.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/94//console This message is automatically generated.
          Hide
          Joep Rottinghuis added a comment -

          No longer relevant for trunk due to Mavenization there.

          Show
          Joep Rottinghuis added a comment - No longer relevant for trunk due to Mavenization there.
          Hide
          Joep Rottinghuis added a comment -

          Will look to see if a different patch is needed for trunk.

          Moving exception stack to a comment to keep description short.
          This is the error I get:

          findbugs:
          [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
          [findbugs] Executing findbugs from ant task
          [findbugs] Running FindBugs...
          [findbugs] Out of memory
          [findbugs] Total memory: 477M
          [findbugs] free memory: 62M
          [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
          ...

          [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
          [findbugs] at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
          [findbugs] at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
          [findbugs] at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
          [findbugs] at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
          ...15 more
          [findbugs] Java Result: 1
          [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
          [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
          [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
          [xslt] : Error! Premature end of file.
          [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
          [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml

          BUILD FAILED
          /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
          at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
          at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
          at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
          at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
          at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
          at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
          ... 15 more
          Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
          at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
          at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
          ... 20 more
          Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
          at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
          at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
          at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
          ... 21 more
          ---------

          etc.etc.
          Total time: 8 minutes 35 seconds
          Build step 'Execute shell' marked build as failure
          [CHECKSTYLE] Collecting checkstyle analysis files...
          CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module with 13525 warnings.
          [FINDBUGS] Skipping publisher since build result is FAILURE
          [WARNINGS] Skipping publisher since build result is FAILURE

          Show
          Joep Rottinghuis added a comment - Will look to see if a different patch is needed for trunk. Moving exception stack to a comment to keep description short. This is the error I get: findbugs: [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs [findbugs] Executing findbugs from ant task [findbugs] Running FindBugs... [findbugs] Out of memory [findbugs] Total memory: 477M [findbugs] free memory: 62M [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar ... [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded [findbugs] at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589) [findbugs] at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565) [findbugs] at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011) [findbugs] at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574) ...15 more [findbugs] Java Result: 1 [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl [xslt] : Error! Premature end of file. [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file. [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml BUILD FAILED /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file. at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313) at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194) at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852) at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) ... 15 more Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file. at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710) ... 20 more Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file. at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446) at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234) at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525) ... 21 more --------- etc.etc. Total time: 8 minutes 35 seconds Build step 'Execute shell' marked build as failure [CHECKSTYLE] Collecting checkstyle analysis files... CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module with 13525 warnings. [FINDBUGS] Skipping publisher since build result is FAILURE [WARNINGS] Skipping publisher since build result is FAILURE
          Hide
          Konstantin Shvachko added a comment -

          Let's keep description small.
          This looks like a good thing to have. I'd recommend to commit this to trunk as well.

          Show
          Konstantin Shvachko added a comment - Let's keep description small. This looks like a good thing to have. I'd recommend to commit this to trunk as well.
          Hide
          Joep Rottinghuis added a comment -

          Once patch is applied and I add the following to my build.properties, the error disappears:

          1. Give FindBugs some more memory due to the large # classes and effort=max parameter.
            findbugs.jvmargs=-Xmx1024M
          Show
          Joep Rottinghuis added a comment - Once patch is applied and I add the following to my build.properties, the error disappears: Give FindBugs some more memory due to the large # classes and effort=max parameter. findbugs.jvmargs=-Xmx1024M
          Hide
          Joep Rottinghuis added a comment -

          Adjusted build file to make FindBugs jvmargs a ant property with the same default value, but this one can be set from outside build.xml

          Show
          Joep Rottinghuis added a comment - Adjusted build file to make FindBugs jvmargs a ant property with the same default value, but this one can be set from outside build.xml
          Hide
          Joep Rottinghuis added a comment -

          Even though this is hardcoded on trunk as well, it is less likely to be a problem due to the project split. After the split each of the three projects will have a much smaller set of files to parse during the FindBugs targets.

          Show
          Joep Rottinghuis added a comment - Even though this is hardcoded on trunk as well, it is less likely to be a problem due to the project split. After the split each of the three projects will have a much smaller set of files to parse during the FindBugs targets.

            People

            • Assignee:
              Joep Rottinghuis
              Reporter:
              Joep Rottinghuis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development