Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6394

[Java] Support conversions between delta vector and partial sum vector

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.16.0
    • Java

    Description

      What is a delta vector/partial sum vector?

      Given an integer vector a with length n, its partial sum vector is another integer vector b with length n + 1, with values defined as:

      b(0) = initial sum
      b(i ) = a(0) + a(1) + ... + a(i - 1) i = 1, 2, ..., n

      Given an integer vector with length n + 1, its delta vector is another integer vector b with length n, with values defined as:

      b(i ) = a(i ) - a(i - 1), i = 0, 1, ... , n -1

      In this issue, we provide utilities to convert between vector and partial sum vector. It is interesting to note that the two operations corresponding to the discrete integration and differentian.

      These conversions have wide applications. For example,

      1. The run-length vector proposed by Micah is based on the partial sum vector, while the deduplication functionality is based on delta vector. This issue provides conversions between them.

      2. The current VarCharVector/VarBinaryVector implementations are based on partial sum vector. We can transform them to delta vectors before IPC, to reduce network traffic.

      3. Converting to delta can be considered as a way for data compression. To further reduce the data volume, the operation can be applied more than once, to further reduce data volume.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            fan_li_ya Liya Fan Assign to me
            fan_li_ya Liya Fan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 1h 50m
              1h 50m

              Issue deployment