Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6685

Improve profile in KrpcDataStreamRecvr and KrpcDataStreamSender

    XMLWordPrintableJSON

Details

    • ghx-label-8

    Description

      The existing profiles in KrpcDataStreamRecvr and KrpcDataStreamSender made it hard to diagnose slow queries shown in IMPALA-6657. In particular, there are times in which the profile of the receiver showing a lot of time waiting for row batches to arrive while the sender is also showing a lot of time waiting for responses of TransmitData() RPC.

      A couple of improvements can be done to make it slightly easier to diagnose the problem:

      • track the number of deferred row batches over time in KrpcDataStreamRecvr
      • track the number of bytes dequeued over time in KrpcDataStreamRecvr
      • track the amount of time row batches spent in deferred queue
      • track the number of bytes sent from KrpcDataStreamSender over time

      The above items help identify cases in which one fragment instances containing an exchange node is slow for a period of time (e.g. the parent of exchange node spills heavily), causing all senders to that fragment instance to block waiting for responses. As all senders are blocked waiting for previous RPC to complete, they will not produce more rows and all other fragment instances will be starved, leading to the high wait time shown in their receiver's profile. The time series counter for the number of deferred row batches in a receiver helps identify cases described above.

      Attachments

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-7449 TotalNetworkThroughput in KrpcDataStreamSender is broken

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-515 add a time-series for bytes sent in the datastreamsender node

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue IMPALA-6692 When partition exchange is followed by sort each sort node becomes a synchronization point across the cluster

          • Critical - Crashes, loss of data, severe memory leak.
          • Reopened

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. IMPALA-10137 Network Debugging / Supportability Improvements

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. IMPALA-5473 Make diagnosing network issues easier

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue IMPALA-10241 Impala Doc: RPC troubleshooting guide

          • Major - Major loss of function.
          • Open

          Activity

            People

              kwho Michael Ho
              kwho Michael Ho
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: