[IMPALA-6609] Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Distributed Exec
• Labels: broken-build
• Target Version: Impala 3.0, Impala 2.12.0
• Epic Color: ghx-label-3
• Assignee: Michael Ho
• Reporter: Michael Ho
• Created: 05/Mar/18 22:17
• Updated: 14/Mar/18 16:11
• Resolved: 13/Mar/18 02:04

Description

The following line in KrpcDataStreamRecvr may lead to use-after-free. A KrpcDataStreamRecvr is co-owned by KrpcDataStreamMgr and an ExchangeNode. There is a window in which the ExchangeNode has already been closed and when the last reference to KrpcDataStreamMgr goes away. In this window, the KrpcDataStreamRecvr queues should all have been cancelled and closed but the receiver itself may still reference resources owned by the ExchangeNode. The general pattern is that once a receiver's queue is cancelled or closed, there should be no more access to its data structures not owned by the receiver or the queue itself. However, there are a couple of places in KrpcDataStreamRecvr which violate this pattern:

void KrpcDataStreamRecvr::RemoveSender(int sender_id) {
  int use_sender_id = is_merging_ ? sender_id : 0;
  sender_queues_[use_sender_id]->DecrementSenders();
  COUNTER_ADD(num_eos_received_, 1); <<-------
}

In the long run, we really need to re-think this business of co-ownership and simplify the lifecycle management of a KrpcDataStreamRecvr object.

The bug above may lead to crash like the following:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00000000018639e9, pid=37254, tid=0x00007f490222b700
#
# JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [impalad+0x14639e9] impala::KrpcDataStreamRecvr::RemoveSender(int)+0x73
#
# Core dump written. Default location: /home/ubuntu/Impala/core or core.37254
#

Issue Links

• relates to: IMPALA-6659 Remove shared ownership of KrpcDataStreamRecvr