Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23744 Reduce query startup latency
  3. HIVE-23745

Avoid copying userpayload in task communicator

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      https://github.com/apache/hive/blob/master/llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java#L182 I see this copy take a few milliseconds sometimes. Delay here adds up for all tasks of a single vertex in LlapTaskCommunicator as it processes tasks one by one. User payload never changes in this codepath. Copy is made because of limitations of Protobuf library. Protobuf adds a UnsafeByteOperations class that avoid copying of ByteBuffers in 3.1 version. This can be resolved when Protobuf is upgraded.

        Attachments

          Activity

            People

            • Assignee:
              mustafaiman Mustafa Iman
              Reporter:
              mustafaiman Mustafa Iman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: