Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6609

Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free

    Details

      Description

      The following line in KrpcDataStreamRecvr may lead to use-after-free. A KrpcDataStreamRecvr is co-owned by KrpcDataStreamMgr and an ExchangeNode. There is a window in which the ExchangeNode has already been closed and when the last reference to KrpcDataStreamMgr goes away. In this window, the KrpcDataStreamRecvr queues should all have been cancelled and closed but the receiver itself may still reference resources owned by the ExchangeNode. The general pattern is that once a receiver's queue is cancelled or closed, there should be no more access to its data structures not owned by the receiver or the queue itself. However, there are a couple of places in KrpcDataStreamRecvr which violate this pattern:

      void KrpcDataStreamRecvr::RemoveSender(int sender_id) {
        int use_sender_id = is_merging_ ? sender_id : 0;
        sender_queues_[use_sender_id]->DecrementSenders();
        COUNTER_ADD(num_eos_received_, 1); <<-------
      }
      

      In the long run, we really need to re-think this business of co-ownership and simplify the lifecycle management of a KrpcDataStreamRecvr object.

      The bug above may lead to crash like the following:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x00000000018639e9, pid=37254, tid=0x00007f490222b700
      #
      # JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
      # Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C [impalad+0x14639e9] impala::KrpcDataStreamRecvr::RemoveSender(int)+0x73
      #
      # Core dump written. Default location: /home/ubuntu/Impala/core or core.37254
      #
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                kwho Michael Ho
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: