Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10263

[C++][Compute] Improve numerical stability of variances merging

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: C++

      Description

      For chunked array, variance kernel needs to merge variances.
      Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
      So current merging method is not stable in extreme cases when chunks are very short and with approximate mean values.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yibo Yibo Cai
                Reporter:
                yibo Yibo Cai
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m