Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9154

KRPC DataStreamService threads blocked in PublishFilter

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 3.4.0
    • Fix Version/s: Impala 3.4.0
    • Component/s: Distributed Exec
    • Labels:

      Description

      I hit this on primitive_many_fragments when doing a single node perf run:

       ./bin/single_node_perf_run.py --num_impalads=1 --scale=30 --ninja --workloads=targeted-perf  --iterations=5
      

      tan

      I noticed that the query was hung and the execution threads were hung sending row batches. Then looking at the RPCz page, all of the threads were busy:

      Multiple threads were stuck in UpdateFilter() - see pstack-exchange.txt. It looks like this is a deadlock bug because a KRPC thread is blocked waiting for an RPC that needs to be served by one of the limited threads from that same thread pool

        Attachments

        1. image-2019-11-13-08-30-27-178.png
          311 kB
          Tim Armstrong
        2. pstack-exchange.txt
          630 kB
          Tim Armstrong

          Issue Links

            Activity

              People

              • Assignee:
                fangyurao Fang-Yu Rao
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: