Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4160 Vectorized Query Execution in Hive
  3. HIVE-4946

Allow prevention of string column re-use for string functions that can set results by reference



    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: vectorization-branch
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      In order to prevent a bug, I had to use BytesColumnVector.setVal instead of BytesColumnVector.setRef when creating the output of all string functions. These include TRIM/LTRIM/RTRIM/SUBSTR, which can be made faster if they can use the setRef method instead. That allows them to avoid the cost of copying the data, by just setting a reference to a string (by setting the byte[] pointer, start, and length).

      As a future performance enhancement, it would be desirable to be able to use setRef for these functions instead of setVal. But to do that, it is necessary to be able to mark a BytesColumnVector that is being referenced into so it is not reclaimed and re-used. So this would require a design and implementation change to the output column manager used in VectorizationContext.

      I'm marking this as "minor" priority because it will result in a small performance enhancement, most likely. It can be deferred for a while.

      The following is an example of a query that will exhibit a bug if setRef is used instead of setVal in the implementation of trim and concat functions:

      select l_shipmode,
      rtrim(concat(l_shipmode,' ')) – incorrect result for this output column
      ,trim(concat(' ',l_shipmode)) – requires this line for bug to show up
      from lineitem_orc
      where l_orderkey = 1;




            • Assignee:
              ehans Eric N. Hanson
            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: