Apache OpenOffice (AOO) Bugzilla – Full Text Issue Listing |
Summary: | Force linear regression curve to go through zero point (origin) or other Y-value defined | ||||||
---|---|---|---|---|---|---|---|
Product: | General | Reporter: | denisl <dlebeuf> | ||||
Component: | chart | Assignee: | AOO issues mailing list <issues> | ||||
Status: | CONFIRMED --- | QA Contact: | |||||
Severity: | Trivial | ||||||
Priority: | P3 | CC: | alek.email, belegdol, issues, jbf.faure, jumbo4444, kamataki, robert.pollak, tony.galmiche.ooo, weigel | ||||
Version: | 3.3.0 or older (OOo) | Keywords: | ms_interoperability, oooqa, rfe_eval_ok | ||||
Target Milestone: | --- | ||||||
Hardware: | All | ||||||
OS: | All | ||||||
Issue Type: | FEATURE | Latest Confirmation in: | --- | ||||
Developer Difficulty: | --- | ||||||
Issue Depends on: | |||||||
Issue Blocks: | 15522 | ||||||
Attachments: |
|
Description
denisl
2004-09-12 20:22:00 UTC
Oups, The numerical example is not very good, X Y 0 0 1 1 2 2.5 3 3 Would be better HI Bettina, one for you. tk reset Target Reassigned to Ingrid. I support this request. As scientifically working student I know that some (linear!) regressions need to go through the origin, by definition. Currently it is not possible to visualize these in charts. Is this already incorporated into the proposal for the new chart engine? One very simple solution is to just add an additional regression type "Linear regression through origin" to the list of available regressions in the data series dialog. We should introduce this missing feature. Even more it should be possible to force the regression line to go through a customizable value - not limitting us to 0. ->bm: Please take care of this issue. "As scientifically working student" you should question if this a valid thing to do. If a physical relationship is known to go through zero and your data does not, you should analyse why that is and write an appropriate conclusion not falsify your results by arbitarily forcing a false fit. One common reason for this effect (apart from experimental error) is that least squares fit assumes that the y residuals are far greater than the x residuals. ie. x is a controlled variable with negligable error s.t. rx/ry -> 0 If this is not the case, the derivation of the least squares line fit is NOT mathematically valid. I have seen examples if dispersed data where the least squares fit was "visibly" wrong by a significant degree. The data did not fit the above requirement and the fit was garbage. Most people are quite ignorant of this important limitation in using linear regression. Least squares can also be strongly influenced by an erroneous data point (a flyer). One further cause is that most implementations of least squares minimise the squares of the y residuals not the perpendiculars to the fitted line. I see little use of this "feature" as suggested. If a user wants to add a fictive slope to his data he can simply add a line close to the lin. regression fit then remove the true line. Adding this as a function will simply mislead non-technical users into thinking that this is an alternative VALID regression fit. It is not. That would seem undersirable. ->oooer: Thanks for you comments. You are probably right, that forcing a regression curve to go through a certain point might lead to wrong results without the user noticing this. Well, we have one vote, one comment here FOR having the feature and one AGAINST. I must admit I do not have too much practical experience with regression, so I cannot really vote for or against this feature. The only point that is a FOR is, that MS has this feature in Excel (that's why this issue has the interoperability keyword), thus we will need this feature for improving our import filters. Of course, we still have to decide about a UI for it. And maybe we can show a warning dialog that tells a user that what he does might not be what he wants. ->oooer: Is there a way to analyze the data and find out if limiting the curve to a certain point makes sense or not? I mean, I could imagine that if your curve passes the point only by a very small amount (relative error), it might not completely fake the result. Of course, the question still remains, why you would want to adjust the curve to pass a certain point. It is only a regression or trend curve, not something where you can read data of. Thanks for your thoughtful comment, oooer. May I suggest following compromise: There SHOULD be a feature to have a graph going through the point of origin, but it should NOT be termed "linear regression", since it is none, as you pointed out. Maybe it could be termed "Best linear fit though point of origin" or anything similar that makes a quick point without being confused with linear regression or other inappropriate statistical terms. (Suggestions welcome.) Adding a line by hand is not a solution at all, since the whole "idea" of the feature is, that the slope is calculated from the data. When drawing the line by hand one would have to calculate the line's angle manually beforehand. Beside that, "drawing" on diagrams is imho never a good idea, since the diagram may change in size and content, so the drawn line wouldn't fit any more after any change. @bm
hmm, I did not realise this was a cloning issue. I recognise the idea of
maximising compatability of import filters but I would have hoped that rather
than being a partial clone of a not-too-recent MSO, things like this would be
where OpenOffice could distinguish itself as a superior product.
The other problem with copying MSO is that users will expect the _same_ false
results or else they will complain the OpenOffice does not work correctly, so
you will need to guess how they force the false zero.
There seem to be too different senarios: the import filter and if/how to add
this functionality to Calc UI.
In the case of the import filter the idea of a dlg is good. I would favour
quite explicit warning that this is included for compatability only and will
produce an invalid fit and incorrect slope. Preferably with a help button
taking them to some text that explains the limitations of lin.regression and
the implications of cheating the fit. Once aware of the choice the user can
decide what to do.
As a feature for Calc UI, I think ANY attempt at lin. regression should post a
warning as to the limitations of the method that will be used and a link to
more info (I see that as the duty of the program since the user cannot know how
it is done and will assume in good faith that the methodology is valid).
@matthias
>>Adding a line by hand is not a solution at all, since the whole "idea" of the
feature is, that the slope is calculated from the data.
That's just the problem, calculating a slope that comes out WROMG is not
calculating from the data but gives the impression it is. It's worse than
having to do a fit by eye because it has a undue bona fide of having been
calculated by a computer.
Maybe it would make more sence to provide an easy means to fit a line by eye
that will follow any resizing etc. in the same way as an applied regression
formula apparently does and then show the formula.
I'm thinking of the way that gimp or blender let you create a line or bezier
curve and pull it around until it fits. At least for the linear case this would
be easy to program. Link it to the chart so that is scales and moves with it.
This will allow those interested in bending the truth to fit thier data to do
what they need to do and also allow a valid means of fitting a line to data
that do not fit the criteria for applying least squares analysis.
I'm not suggesting we make the use get out pen and paper here. The human brain
is much better at seeing the "best fit" than an incorrectly applied bit of
maths.
I cannot even imagine the ammount of innocently derived yet spurious
"scientific results" caused by inappropriate use of linear regression. One
major cause of this (appart from the quality of science education) is
spreadsheet software that makes it as easy as clicking a button.
I think OpenOffice has a chance to innovate rather than follow here.
Thanks for your interest in my comments.
>> since the diagram may change in size and content, so the drawn line wouldn't
>> fit any more after any change.
@matthias
sorry, I did not understand your point until I reread this.
If a user fits a line by eye , he will know it is no longer valid if he changes
the data.
If he sets an automatic regression to inappropriate data it will never fit but
he will believe it does.
The former would seem to be the better choice.
@oooer:
You see this from a strict scientifically point of view when you say "bend the
truth". I may point out that Office suites are used (and are to be used) by all
sorts of people, some of which might not even care about scientific statistics,
just want a nice fitting for visualization.
Take f.e. a businessman that wants to visualize the mean growth of their
company's profits and which knows that it had 0 profits in, lets say, 2000,
because it the company was just founded this year. He/she wants a diagram with a
line going through the "(2000, 0)" point simply for the look of it.
Also sometimes constraints such as "fixed points" simply exist. Sometimes it is
better for a computer model to prefer a slightly wrong relationship formula over
the correct regression, simply to avoid computation side effects and artefacts,
or to simplify calculation. You'd maybe call this unscientifically, but such
tradeoffs are imho justified in science - as long as they are documented.
(I won't go deeper in this - this mailing list is the wrong place for such debates.)
Specifically to your comments:
- Having warning dialogs if the user does something "unscienfically" would be
very appropriate in a scientific software like an actual statistics program. For
a "general" office suite I find this out of scope. Imho OOo cannot (and should
not even try) to find out when, lets say, diagramm types are inappropriate,
diagram colors are hard to tell apart, formulae are nonsense or anything of this
sort.
> - "The human brain is much better at seeing the "best fit" than an incorrectly
> applied bit of maths."
You seem to have a much better brain for this, since with my brain I would not
dare to fit a line manually, except for a rough sketch. ;-)
But seriously, people don't use OOo's diagram features to "paint". And I don't
like this "penalty" approach: "If you are not doing strict statistics, use OOo
Draw and paint your line by hand."
*** Issue 71191 has been marked as a duplicate of this issue. *** Any updates on this? REGLINP already supports a fixed intercept, so... Umm, I mean LINEST. /me loves translating the functions. change owner I agree. This is very important function for me as a science student. I can't stop using Microsoft Office until I can do this in Open Office - at least for linear regression. In fact in the interest of flexibility I think you should be able to choose exactly where it intersects either axis. I know there are scientific arguments for and against this function but what difference does it make who is scientifically right or wrong. If this is a function that users want to use then Open Office should be able to do it as simply and quickly as possible. Its also necessary to be able to extrapolate the line as much as desired in either direction. reset to new *** Issue 96461 has been marked as a duplicate of this issue. *** There are many comments about this not being important as it is "faking data". That simply isn't the case. In the real world and not just on paper, there are error tolerances in what can be measured and they are often not exact. This is why you need a trendline in the first place. If you have only a small number of points there is a higher probability of the fit not meeting reality. Now, those who suggest that the data really does go through a different y intercept there are examples where it simply must be through a given point, such as zero. For example, if you plot concentration versus light absorbance then zero concentration must have zero absorbance. Extension of a spring might be another example. If there is no mass on the spring it needs to have a zero extension. The data in these examples would be MORE incorrect to have them pass through a y intercept other than zero. We should not only be able to force the regression curve to go through zero, but also through any other definable y axis intercept. I will attach a screenshot from main competitor as an example (sorry only in german). Created attachment 58788 [details]
Screenshot from competitor. Define y axis intercept in options dialog of regression curve.
I have used OO for a number of years, and this feature is one I Keep missing every so often. So please find the resources to implement this feature. Same as shamran (and totally agree with dskene's post). So UP! Hello, I am pleased to inform you that Marcin Gutman has build an extension based on Laurent Godard's macro CorelPoly. It can be downloaded at http://extensions.services.openoffice.org/node/4387? It is not the solution of this issue but it looks, from my point of view, has a good workaround. In addition to polynomial regression, it also enable "force intercept" option. Please add your comments on this extension NOT HERE, but on extension web-page. Comments and users feedback are welcome. Next versions will include more robust method thanks to LINEST function, and some translation. Hello, CorelPolyGUI extension has been updated http://extensions.services.openoffice.org/en/node/4634 It contains more robust method (QR decomposition), a simpler way to use "force intercept" in case of linear problem, and translation in French and Polish. Sorry for the broken link. It is better to use: http://extensions.services.openoffice.org/en/project/CorelPolyGUI Latest version 0.3.2 corrects minor bugs. Hello, New version of CorelPoly (v 1.0!) available on http://extensions-test.libreoffice.org/extension-center/improved-trend-lines/ and soon on OOo repository. Now all classic regression are included, with force intercept option. *** Issue 97439 has been marked as a duplicate of this issue. *** |