Commons Math
  1. Commons Math
  2. MATH-784

Javadoc of AbstractLeastSquaresOptimizer.guessParametersErrors() is too vague

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0
    • Fix Version/s: 3.3

      Description

      This bug report follows a recent discussion available here. It is now recognized that the values returned by guessParametersErrors() are in fact known as (asymptotic) standard errors. The javadoc should be made more explicit. Besides, the values returned by this method should be tested. The reference datasets from NIST are to be used.

      1. SyntheticData.java
        6 kB
        Sébastien Brisard
      2. SyntheticDataLinear.java
        7 kB
        Sébastien Brisard
      3. AbstractLeastSquaresOptimizerTestValidation.java
        11 kB
        Gilles
      4. StraightLineProblem.java
        5 kB
        Gilles
      5. RandomStraightLinePointGenerator.java
        3 kB
        Gilles
      6. OUT.10
        0.4 kB
        Gilles
      7. OUT.100
        0.4 kB
        Gilles
      8. montecarlo_params.eps
        62 kB
        Gilles

        Issue Links

          Activity

          Hide
          Sébastien Brisard added a comment -

          Made improvements to the javadoc in r1332076.

          Show
          Sébastien Brisard added a comment - Made improvements to the javadoc in r1332076 .
          Hide
          Sébastien Brisard added a comment -

          The Statistical Reference Datasets from NIST provide reference values of the standard deviation of the parameters for many datasets. In r1332086 these values are used to test guessParametersErrors().

          Show
          Sébastien Brisard added a comment - The Statistical Reference Datasets from NIST provide reference values of the standard deviation of the parameters for many datasets. In r1332086 these values are used to test guessParametersErrors() .
          Hide
          Gilles added a comment -

          Did you confirm that the correct "error" is returned by "guessParametersErrors" but not by the diagonal elements of the covariance matrix?

          Show
          Gilles added a comment - Did you confirm that the correct "error" is returned by "guessParametersErrors" but not by the diagonal elements of the covariance matrix?
          Hide
          Sébastien Brisard added a comment - - edited

          I did confirm that guessParametersErrors() in fact returns the same estimate of the standard deviation of the parameters as the one listed in the NIST data.

          Show
          Sébastien Brisard added a comment - - edited I did confirm that guessParametersErrors() in fact returns the same estimate of the standard deviation of the parameters as the one listed in the NIST data.
          Hide
          Sébastien Brisard added a comment -

          Attached is a Monte-Carlo simulation where 100000 realizations of the same dataset are sampled. This leads to a series of 100,000 optimized values of the parameters of the model. Then, the standard deviation of these parameters can be estimated, and compared to the value returned by guessParametersErrors(). The match is very good. However, the match with the sqrt of the diagonal elements of the covariance matrix is also very good. Thinking about it, it's not a big surprise, since the two estimates differ only by a factor sqrt(n/(n-m)), where n is the number of observations, and m the number of parameters (if the fit is good, the optimum value of chi2 is nearly equal to n).
          Also shown by this program: the sqrt of the diag coeffs of the covariance matrix provide the 68% confidence interval on the parameters.

          What remains to be explored

          • use smaller number of observables (then n/(n-m) should be significantly greater than 1),
          • use observations which are not distributed according to a gaussian law.
          Show
          Sébastien Brisard added a comment - Attached is a Monte-Carlo simulation where 100000 realizations of the same dataset are sampled. This leads to a series of 100,000 optimized values of the parameters of the model. Then, the standard deviation of these parameters can be estimated, and compared to the value returned by guessParametersErrors() . The match is very good. However, the match with the sqrt of the diagonal elements of the covariance matrix is also very good. Thinking about it, it's not a big surprise, since the two estimates differ only by a factor sqrt(n/(n-m)) , where n is the number of observations, and m the number of parameters (if the fit is good, the optimum value of chi2 is nearly equal to n ). Also shown by this program: the sqrt of the diag coeffs of the covariance matrix provide the 68% confidence interval on the parameters. What remains to be explored use smaller number of observables (then n/(n-m) should be significantly greater than 1), use observations which are not distributed according to a gaussian law.
          Hide
          Sébastien Brisard added a comment -

          Attached a new version of the MC simulation, whith a linear model y = a[0] + a[1] * x, and only 3 observation points x[0] = 0.333, x[1] = 0.666 and x[2] = 1.0. It looks like in that case, the two estimator differ quite significantly, and in fact, the simple one (unweighted square root of the diagonal coefficients) is a much better estimator of the sd on the parameters.

          It comes out as a surprise, because it contradicts formulas (34) and (35) in MathWorld. Also, the standard deviation on the parameters in NIST StRD is computed with the other formula.

          Show
          Sébastien Brisard added a comment - Attached a new version of the MC simulation, whith a linear model y = a [0] + a [1] * x, and only 3 observation points x [0] = 0.333, x [1] = 0.666 and x [2] = 1.0. It looks like in that case, the two estimator differ quite significantly, and in fact, the simple one (unweighted square root of the diagonal coefficients) is a much better estimator of the sd on the parameters. It comes out as a surprise, because it contradicts formulas (34) and (35) in MathWorld . Also, the standard deviation on the parameters in NIST StRD is computed with the other formula.
          Hide
          Gilles added a comment -

          Here are the code I was talking about on the ML.
          You have been doing the same thing it seems...

          Show
          Gilles added a comment - Here are the code I was talking about on the ML. You have been doing the same thing it seems...
          Hide
          Gilles added a comment -

          The "OUT.10" and "OUT.100" are the output of the first unit test in "AbstractLeastSquaresOptimizerTestValidation.java" for 10 and 100 observations respectively.
          The Monte-Carlo (on the observations) confirms that "sigma" (from the covariance matrix) better approximates the standard deviation (of the parameter distribution generated by the Monte-Carlo).

          Show
          Gilles added a comment - The "OUT.10" and "OUT.100" are the output of the first unit test in "AbstractLeastSquaresOptimizerTestValidation.java" for 10 and 100 observations respectively. The Monte-Carlo (on the observations) confirms that "sigma" (from the covariance matrix) better approximates the standard deviation (of the parameter distribution generated by the Monte-Carlo).
          Hide
          Gilles added a comment -

          "montecarlo_params.eps" is the plot of a Monte-Carlo run on the parameters, i.e. generating sets of parameters, and recording which (plotted in green) have a chi-square lower than "1 + chi2_b", where "chi2_b" is the chi-square of the optimal solution (plotted in red).

          Show
          Gilles added a comment - "montecarlo_params.eps" is the plot of a Monte-Carlo run on the parameters , i.e. generating sets of parameters, and recording which (plotted in green) have a chi-square lower than "1 + chi2_b", where "chi2_b" is the chi-square of the optimal solution (plotted in red).
          Hide
          Sébastien Brisard added a comment -

          It seems we did the same simulations, with same conclusions! Good
          OK, I think I did a lot of noise for nothing, it seems, and I do apologize for that.
          I just want to understand though, why the fomulas in MathWorld and NIST Statistical Reference Datasets are different.

          Show
          Sébastien Brisard added a comment - It seems we did the same simulations, with same conclusions! Good OK, I think I did a lot of noise for nothing, it seems, and I do apologize for that. I just want to understand though, why the fomulas in MathWorld and NIST Statistical Reference Datasets are different.
          Hide
          Sébastien Brisard added a comment -

          In this thread, it was agreed to

          • deprecate guessParametersErrors()
          • create a new method, namely getSigma(), which simply returns the square root of the diagonal coefficients of the covariance matrix. If necessary, the values previously returned by guessParametersErrors() can easily be retrieved from getSigma() and getChiSquare().

          The rationale for this decision is copied below from the mailing list

          Independently of the explanation to be provided by Dimitri, I think that
          there are code design arguments in favour of deprecating (and later,
          deleting) the "guessParametersErrors" method, as follows.

          In the context of the "optimization.general" package, one assumes that a
          Jacobian matrix is available. From there, the code in "AbstractLeastSquares"
          computes the covariance matrix, from which one can readily extract the
          "sigma".
          This can be done without computing the chi-square! [While, as you have
          probably noticed, the "guessParametersErrors" will not behave nicely if you
          don't call "updateResidualsAndCost()" beforehand.]

          For the class to be self-consistent, the story can end here: Any additional
          utilities can lead to wrong expectations from different types of users (as
          we've demonstrated here).
          Indeed, confidence intervals refer to additional variables (as Dimitri
          wrote: "By how much can a parameter change before the normalized chi2
          changes by <some number>?"). Being able to answer those questions also
          involves the correlations between the parameters (cf. the plot I've attached
          to MATH-784), whereas "guessParametersErrors" does not take them into
          account.

          This was done in r1334315.

          Show
          Sébastien Brisard added a comment - In this thread , it was agreed to deprecate guessParametersErrors() create a new method, namely getSigma() , which simply returns the square root of the diagonal coefficients of the covariance matrix. If necessary, the values previously returned by guessParametersErrors() can easily be retrieved from getSigma() and getChiSquare() . The rationale for this decision is copied below from the mailing list Independently of the explanation to be provided by Dimitri, I think that there are code design arguments in favour of deprecating (and later, deleting) the "guessParametersErrors" method, as follows. In the context of the "optimization.general" package, one assumes that a Jacobian matrix is available. From there, the code in "AbstractLeastSquares" computes the covariance matrix, from which one can readily extract the "sigma". This can be done without computing the chi-square! [While, as you have probably noticed, the "guessParametersErrors" will not behave nicely if you don't call "updateResidualsAndCost()" beforehand.] For the class to be self-consistent, the story can end here: Any additional utilities can lead to wrong expectations from different types of users (as we've demonstrated here). Indeed, confidence intervals refer to additional variables (as Dimitri wrote: "By how much can a parameter change before the normalized chi2 changes by <some number>?"). Being able to answer those questions also involves the correlations between the parameters (cf. the plot I've attached to MATH-784 ), whereas "guessParametersErrors" does not take them into account. This was done in r1334315 .
          Hide
          Sébastien Brisard added a comment -


          As far as version 3.1 is concerned, this issue is fixed. However, this ticket must remain open until 4.0 is on the tracks, as some deprecated methods must be removed.

          Show
          Sébastien Brisard added a comment - As far as version 3.1 is concerned, this issue is fixed . However, this ticket must remain open until 4.0 is on the tracks, as some deprecated methods must be removed.
          Hide
          Gilles added a comment -

          Method referred to in the description only exists in deprecated classes.

          Show
          Gilles added a comment - Method referred to in the description only exists in deprecated classes.
          Hide
          Luc Maisonobe added a comment -

          Closing all resolved issue now available in released 3.3 version.

          Show
          Luc Maisonobe added a comment - Closing all resolved issue now available in released 3.3 version.

            People

            • Assignee:
              Unassigned
              Reporter:
              Sébastien Brisard
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development