Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-469

Add EncodedStringVectorBatch to expose string dictionary in VectorBatch

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.5, 1.6.0, 2.0.0
    • Fix Version/s: None
    • Component/s: Reader
    • Labels:
      None

      Description

      Propose to add EncodedStringVectorBatch to expose string dictionary in VectorBatch. Exposing string dictionary will benefits as bellow:

      • Enable computation over encoded data. By exposing dictionary in vector batch we would be able to implement filter operator in a more efficient way. In our POC, by enabling encoded data based filter, we achieved 8% E2E perf improvement on tpch q1
      • Make data serialization more efficient. Currently when serializing Orc Vectorbatch, we have to copy all the strings in a vector even though the string data is already dictionary encoded. Exposing String Dictionary will enable vector batch serializer to remove unnecessary string memcpy, which will greatly improve serialization efficiency.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Yurui Zhou Yurui Zhou
              Reporter:
              Yurui Zhou Yurui Zhou

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                  Issue deployment