Commons Math
  1. Commons Math
  2. MATH-1057

BOBYQAOptimizerTest has two failing tests

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.2
    • Fix Version/s: 3.3
    • Labels:
      None
    • Environment:

      Mac OS X 10.9 and also Linux 3.4 kernel; Java 7; Maven 3.1.1

      Description

      I see two test failures, in both the copies of BOBYQAOptimizerTest:

      Failed tests: 
        BOBYQAOptimizerTest.testAckley:209->doTest:282->doTest:338 expected:<0.0> but was:<1.047765607609108E-8>
        BOBYQAOptimizerTest.testAckley:208->doTest:281->doTest:336 expected:<0.0> but was:<1.047765607609108E-8>
      
      Tests in error: 
        BOBYQAOptimizerTest.testDiffPow:187->doTest:282->doTest:322 » TooManyEvaluations
        BOBYQAOptimizerTest.testDiffPow:186->doTest:281->doTest:326 » TooManyEvaluations
      

      (This predated the patches I've worked on so I don't think it's me!)

      I tried on Mac OS X and Linux and see the same, so don't think it is an environment issue. I'll see if a little digging can uncover the issue from a recent commit.

        Issue Links

          Activity

          Hide
          Luc Maisonobe added a comment -

          Closing all resolved issue now available in released 3.3 version.

          Show
          Luc Maisonobe added a comment - Closing all resolved issue now available in released 3.3 version.
          Hide
          Sean Owen added a comment -

          Yeah I see the point but the artifact is now called 'math3' as it was not backwards compatible with 2.x and was sometimes necessary to deploy together. So it really is math3 3.2.

          Show
          Sean Owen added a comment - Yeah I see the point but the artifact is now called 'math3' as it was not backwards compatible with 2.x and was sometimes necessary to deploy together. So it really is math3 3.2.
          Hide
          Emmanuel Bourg added a comment -

          It might be a good idea to rename the source archive for the next releases to commons-math-3.x-bin.tar.gz, that's indeed confusing.

          Show
          Emmanuel Bourg added a comment - It might be a good idea to rename the source archive for the next releases to commons-math-3.x-bin.tar.gz, that's indeed confusing.
          Hide
          Gwendal added a comment -

          Oh, right! I misread 3.3 instead of 3-3... My apologies! I will try to be more careful in the future.

          Thanks a lot for answering this, and sorry again!

          Show
          Gwendal added a comment - Oh, right! I misread 3.3 instead of 3-3... My apologies! I will try to be more careful in the future. Thanks a lot for answering this, and sorry again!
          Hide
          Sean Owen added a comment -

          Yes, but you show you are working with 3.2. Look at what you downloaded.

          Show
          Sean Owen added a comment - Yes, but you show you are working with 3.2. Look at what you downloaded.
          Hide
          Gwendal added a comment -

          I've simply downloaded this source archive : http://apache.mirrors.multidist.eu//commons/math/binaries/commons-math3-3.2-bin.tar.gz

          Then I tried to compile it with Maven, and I obtained these errors.

          I thought the problem was (theoretically) fixed in 3.3. Maybe I misunderstood?

          Show
          Gwendal added a comment - I've simply downloaded this source archive : http://apache.mirrors.multidist.eu//commons/math/binaries/commons-math3-3.2-bin.tar.gz Then I tried to compile it with Maven, and I obtained these errors. I thought the problem was (theoretically) fixed in 3.3. Maybe I misunderstood?
          Hide
          Sean Owen added a comment -

          The error is indeed reported against 3.2 and fixed for 3.3. Are you saying you see this in HEAD?

          Show
          Sean Owen added a comment - The error is indeed reported against 3.2 and fixed for 3.3. Are you saying you see this in HEAD?
          Hide
          Gwendal added a comment -

          Hello,

          Sorry to post a message on this Resolved bug, but I appear to have the exact same problem with commons-math3-3.2. Here are the interesting parts of my Maven output:
          ------------------------------------------------------------------------------------------
          testDiffPow(org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest) Time elapsed: 2.662 sec <<< ERROR!
          org.apache.commons.math3.exception.TooManyEvaluationsException: illegal state: maximal count (12,000) exceeded: evaluations
          at org.apache.commons.math3.optim.BaseOptimizer$MaxEvalCallback.trigger(BaseOptimizer.java:213)
          at org.apache.commons.math3.util.Incrementor.incrementCount(Incrementor.java:156)
          at org.apache.commons.math3.optim.BaseOptimizer.incrementEvaluationCount(BaseOptimizer.java:162)
          at org.apache.commons.math3.optim.nonlinear.scalar.MultivariateOptimizer.computeObjectiveValue(MultivariateOptimizer.java:115)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.bobyqb(BOBYQAOptimizer.java:823)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.bobyqa(BOBYQAOptimizer.java:329)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:241)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:49)
          at org.apache.commons.math3.optim.BaseOptimizer.optimize(BaseOptimizer.java:143)
          at org.apache.commons.math3.optim.BaseMultivariateOptimizer.optimize(BaseMultivariateOptimizer.java:66)
          at org.apache.commons.math3.optim.nonlinear.scalar.MultivariateOptimizer.optimize(MultivariateOptimizer.java:64)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:322)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:282)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.testDiffPow(BOBYQAOptimizerTest.java:187)

          testDiffPow(org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest) Time elapsed: 2.907 sec <<< ERROR!
          org.apache.commons.math3.exception.TooManyEvaluationsException: illegal state: maximal count (12,000) exceeded: evaluations
          at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.computeObjectiveValue(BaseAbstractMultivariateOptimizer.java:108)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.bobyqb(BOBYQAOptimizer.java:828)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.bobyqa(BOBYQAOptimizer.java:334)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:246)
          at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.optimizeInternal(BaseAbstractMultivariateOptimizer.java:206)
          at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.optimize(BaseAbstractMultivariateOptimizer.java:145)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:326)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:281)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.testDiffPow(BOBYQAOptimizerTest.java:186)

          testAckley(org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest) Time elapsed: 0.025 sec <<< FAILURE!
          java.lang.AssertionError: expected:<0.0> but was:<1.047765607609108E-8>
          at org.junit.Assert.fail(Assert.java:93)
          at org.junit.Assert.failNotEquals(Assert.java:647)
          at org.junit.Assert.assertEquals(Assert.java:443)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:336)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:281)
          at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.testAckley(BOBYQAOptimizerTest.java:208)

          testAckley(org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest) Time elapsed: 0.025 sec <<< FAILURE!
          java.lang.AssertionError: expected:<0.0> but was:<1.047765607609108E-8>
          at org.junit.Assert.fail(Assert.java:93)
          at org.junit.Assert.failNotEquals(Assert.java:647)
          at org.junit.Assert.assertEquals(Assert.java:443)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:338)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:282)
          at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.testAckley(BOBYQAOptimizerTest.java:209)

          Failed tests:
          BOBYQAOptimizerTest.testAckley:208->doTest:281->doTest:336 expected:<0.0> but was:<1.047765607609108E-8>
          BOBYQAOptimizerTest.testAckley:209->doTest:282->doTest:338 expected:<0.0> but was:<1.047765607609108E-8>

          Tests in error:
          BOBYQAOptimizerTest.testDiffPow:186->doTest:281->doTest:326 » TooManyEvaluations
          BOBYQAOptimizerTest.testDiffPow:187->doTest:282->doTest:322 » TooManyEvaluations
          ------------------------------------------------------------------------------------------

          I am using Fedora 20 x86_64 with kernel 3.12.7 and with java-1.7.0-openjdk-1.7.0.60-2.4.4.1.fc20.x86_64.

          Show
          Gwendal added a comment - Hello, Sorry to post a message on this Resolved bug, but I appear to have the exact same problem with commons-math3-3.2. Here are the interesting parts of my Maven output: ------------------------------------------------------------------------------------------ testDiffPow(org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest) Time elapsed: 2.662 sec <<< ERROR! org.apache.commons.math3.exception.TooManyEvaluationsException: illegal state: maximal count (12,000) exceeded: evaluations at org.apache.commons.math3.optim.BaseOptimizer$MaxEvalCallback.trigger(BaseOptimizer.java:213) at org.apache.commons.math3.util.Incrementor.incrementCount(Incrementor.java:156) at org.apache.commons.math3.optim.BaseOptimizer.incrementEvaluationCount(BaseOptimizer.java:162) at org.apache.commons.math3.optim.nonlinear.scalar.MultivariateOptimizer.computeObjectiveValue(MultivariateOptimizer.java:115) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.bobyqb(BOBYQAOptimizer.java:823) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.bobyqa(BOBYQAOptimizer.java:329) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:241) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:49) at org.apache.commons.math3.optim.BaseOptimizer.optimize(BaseOptimizer.java:143) at org.apache.commons.math3.optim.BaseMultivariateOptimizer.optimize(BaseMultivariateOptimizer.java:66) at org.apache.commons.math3.optim.nonlinear.scalar.MultivariateOptimizer.optimize(MultivariateOptimizer.java:64) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:322) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:282) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.testDiffPow(BOBYQAOptimizerTest.java:187) testDiffPow(org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest) Time elapsed: 2.907 sec <<< ERROR! org.apache.commons.math3.exception.TooManyEvaluationsException: illegal state: maximal count (12,000) exceeded: evaluations at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.computeObjectiveValue(BaseAbstractMultivariateOptimizer.java:108) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.bobyqb(BOBYQAOptimizer.java:828) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.bobyqa(BOBYQAOptimizer.java:334) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizer.doOptimize(BOBYQAOptimizer.java:246) at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.optimizeInternal(BaseAbstractMultivariateOptimizer.java:206) at org.apache.commons.math3.optimization.direct.BaseAbstractMultivariateOptimizer.optimize(BaseAbstractMultivariateOptimizer.java:145) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:326) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:281) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.testDiffPow(BOBYQAOptimizerTest.java:186) testAckley(org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest) Time elapsed: 0.025 sec <<< FAILURE! java.lang.AssertionError: expected:<0.0> but was:<1.047765607609108E-8> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:443) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:336) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:281) at org.apache.commons.math3.optimization.direct.BOBYQAOptimizerTest.testAckley(BOBYQAOptimizerTest.java:208) testAckley(org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest) Time elapsed: 0.025 sec <<< FAILURE! java.lang.AssertionError: expected:<0.0> but was:<1.047765607609108E-8> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:443) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:338) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.doTest(BOBYQAOptimizerTest.java:282) at org.apache.commons.math3.optim.nonlinear.scalar.noderiv.BOBYQAOptimizerTest.testAckley(BOBYQAOptimizerTest.java:209) Failed tests: BOBYQAOptimizerTest.testAckley:208->doTest:281->doTest:336 expected:<0.0> but was:<1.047765607609108E-8> BOBYQAOptimizerTest.testAckley:209->doTest:282->doTest:338 expected:<0.0> but was:<1.047765607609108E-8> Tests in error: BOBYQAOptimizerTest.testDiffPow:186->doTest:281->doTest:326 » TooManyEvaluations BOBYQAOptimizerTest.testDiffPow:187->doTest:282->doTest:322 » TooManyEvaluations ------------------------------------------------------------------------------------------ I am using Fedora 20 x86_64 with kernel 3.12.7 and with java-1.7.0-openjdk-1.7.0.60-2.4.4.1.fc20.x86_64.
          Hide
          Thomas Neidhart added a comment -

          Fixed in r1540075.

          Thanks for your patch!
          For now I did only update the test classes as this seems to be sufficient.
          I created another issue to replace calls to Math.xxx with calls to FastMath.xxx throughout CM.

          Show
          Thomas Neidhart added a comment - Fixed in r1540075. Thanks for your patch! For now I did only update the test classes as this seems to be sufficient. I created another issue to replace calls to Math.xxx with calls to FastMath.xxx throughout CM.
          Hide
          Gilles added a comment -

          The extreme sensitivity of some tests was noticed quite some time ago. The main problem is that we introduced this algorithm into CM although the code was nowhere near to something a Java programmer can understand. And this was already after I performed extensive work to modify the code that had been auto-generated from the original FORTRAN implementation.
          Further code readability improvements were stalled due to
          tests failing after seemingly innocuous changes; hence the need for expert advice in order to know what is actually to be expected from the tests and by how much the tolerance can be lowered (while still retaining the ability to catch erroneous changes during the code rewrite).

          Show
          Gilles added a comment - The extreme sensitivity of some tests was noticed quite some time ago. The main problem is that we introduced this algorithm into CM although the code was nowhere near to something a Java programmer can understand. And this was already after I performed extensive work to modify the code that had been auto-generated from the original FORTRAN implementation. Further code readability improvements were stalled due to tests failing after seemingly innocuous changes; hence the need for expert advice in order to know what is actually to be expected from the tests and by how much the tolerance can be lowered (while still retaining the ability to catch erroneous changes during the code rewrite).
          Hide
          Sean Owen added a comment -

          For the record, my java versions:

          OS X:

          java version "1.7.0_45"
          Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
          Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
          

          Linux:

          java version "1.7.0_45"
          OpenJDK Runtime Environment (amzn-2.4.3.2.32.amzn1-x86_64 u45-b15)
          OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
          
          Show
          Sean Owen added a comment - For the record, my java versions: OS X: java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) Linux: java version "1.7.0_45" OpenJDK Runtime Environment (amzn-2.4.3.2.32.amzn1-x86_64 u45-b15) OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)
          Hide
          Thomas Neidhart added a comment -

          Ok I understand now.
          The difference was happening before, when we used Math.xxx calls, which returned different results for different jdk versions.

          Now, when using FastMath, the result is of course consistent with all jdks, as FastMath is a pure java implementation. The result for this test has change slightly and a epsilon of 1e-7 should be used to compensate the for change imho.

          @Gilles: in case you read here, as you have done most of the work on this optimizer, do you think that the switch from Math to FastMath is ok, or was there a specific reason why Math was used in this case?

          Show
          Thomas Neidhart added a comment - Ok I understand now. The difference was happening before, when we used Math.xxx calls, which returned different results for different jdk versions. Now, when using FastMath, the result is of course consistent with all jdks, as FastMath is a pure java implementation. The result for this test has change slightly and a epsilon of 1e-7 should be used to compensate the for change imho. @Gilles: in case you read here, as you have done most of the work on this optimizer, do you think that the switch from Math to FastMath is ok, or was there a specific reason why Math was used in this case?
          Hide
          Thomas Neidhart added a comment -

          Ah ok, now I get the same error, strange that I did not see it before.
          In this case I think it would be fine to change the epsilon to 1e-7 but I would like to understand why we get different results for jdk 1.5 and 1.7 here.

          Show
          Thomas Neidhart added a comment - Ah ok, now I get the same error, strange that I did not see it before. In this case I think it would be fine to change the epsilon to 1e-7 but I would like to understand why we get different results for jdk 1.5 and 1.7 here.
          Hide
          Thomas Neidhart added a comment -

          Yes, but what exact jvm?

          I use the following:

          java version "1.7.0_25"
          OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2)
          OpenJDK Server VM (build 23.7-b01, mixed mode)

          Show
          Thomas Neidhart added a comment - Yes, but what exact jvm? I use the following: java version "1.7.0_25" OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.12.04.2) OpenJDK Server VM (build 23.7-b01, mixed mode)
          Hide
          Sean Owen added a comment -

          Oh I see – see above, Java 7 on OS X 10.9 and on Linux.

          Show
          Sean Owen added a comment - Oh I see – see above, Java 7 on OS X 10.9 and on Linux.
          Hide
          Thomas Neidhart added a comment -

          With all my different jdks I used to test the problems, I only had failures with the testDiffPow and testsDiffPow methods, never with the testAckley.

          What is your environment?

          Show
          Thomas Neidhart added a comment - With all my different jdks I used to test the problems, I only had failures with the testDiffPow and testsDiffPow methods, never with the testAckley. What is your environment?
          Hide
          Sean Owen added a comment -

          Here's a patch implementing your change. Yes it fixes the failure for testDiffPow for me too and sounds like a good change. Not sure about the other. Maybe the tolerance needs to be loosened?

          Show
          Sean Owen added a comment - Here's a patch implementing your change. Yes it fixes the failure for testDiffPow for me too and sounds like a good change. Not sure about the other. Maybe the tolerance needs to be loosened?
          Hide
          Thomas Neidhart added a comment -

          For the testDiffPow testcase, I further debugged with the two different jdks, and after iteration 2219 the objective function output changes.

          The difference/error then accumulates and results that the algorithm converges much slower. I could track down the problem to the Math implementation, most likely the sqrt or exp. Changing all the Math calls to our own FastMath solves the problem and the test executes correctly for all jdks.

          I wonder why we use Math in the first place, as normally we eat our own dog-food and use FastMath.

          btw. the problem also is existent in the 3.2 release so not at all related to your previous patch.

          Show
          Thomas Neidhart added a comment - For the testDiffPow testcase, I further debugged with the two different jdks, and after iteration 2219 the objective function output changes. The difference/error then accumulates and results that the algorithm converges much slower. I could track down the problem to the Math implementation, most likely the sqrt or exp. Changing all the Math calls to our own FastMath solves the problem and the test executes correctly for all jdks. I wonder why we use Math in the first place, as normally we eat our own dog-food and use FastMath. btw. the problem also is existent in the 3.2 release so not at all related to your previous patch.
          Hide
          Thomas Neidhart added a comment -

          Tests fail when run with jdk 1.5 in my environment.

          Running them with java 1.6 or 1.7 is successful.

          Show
          Thomas Neidhart added a comment - Tests fail when run with jdk 1.5 in my environment. Running them with java 1.6 or 1.7 is successful.
          Hide
          Sean Owen added a comment -

          Hmm, I show that this fails even from the first time the test was added in r1420684. Anyone else seeing the same? if not, what platform I wonder?

          Show
          Sean Owen added a comment - Hmm, I show that this fails even from the first time the test was added in r1420684. Anyone else seeing the same? if not, what platform I wonder?

            People

            • Assignee:
              Unassigned
              Reporter:
              Sean Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development