Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-469

Add EncodedStringVectorBatch to expose string dictionary in VectorBatch

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.5, 1.6.0, 2.0.0
    • None
    • Reader
    • None

    Description

      Propose to add EncodedStringVectorBatch to expose string dictionary in VectorBatch. Exposing string dictionary will benefits as bellow:

      • Enable computation over encoded data. By exposing dictionary in vector batch we would be able to implement filter operator in a more efficient way. In our POC, by enabling encoded data based filter, we achieved 8% E2E perf improvement on tpch q1
      • Make data serialization more efficient. Currently when serializing Orc Vectorbatch, we have to copy all the strings in a vector even though the string data is already dictionary encoded. Exposing String Dictionary will enable vector batch serializer to remove unnecessary string memcpy, which will greatly improve serialization efficiency.

      Attachments

        Issue Links

          Activity

            People

              Yurui Zhou Yurui Zhou
              Yurui Zhou Yurui Zhou
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m