[IMPALA-2314] Crash in PHJ::PrepareNextPartition() in call to LargestSpilledPartition()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.1, Impala 2.2
• Fix Version/s: Impala 2.3.0, Impala 2.2.8
• Labels: crash resource-management
• Target Version: Impala 2.3.0, Impala 2.2.8
• Assignee: Ippokratis Pandis
• Reporter: Ippokratis Pandis
• Created: 09/Sep/15 20:28
• Updated: 20/Dec/15 00:05
• Resolved: 18/Sep/15 14:18

Description

We had two reports of the following crash.

The callstack:

#0  0x0000003144e32625 in raise () from /lib64/libc.so.6
#1  0x0000003144e33e05 in abort () from /lib64/libc.so.6
#2  0x00007f984e5cda55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3  0x00007f984e74df87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4  0x00007f984e5d296f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x0000000000c15c2e in impala::PartitionedHashJoinNode::LargestSpilledPartition (this=0x7f9218ca5400)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/exec/partitioned-hash-join-node.cc:719
#7  0x0000000000c1f08b in impala::PartitionedHashJoinNode::PrepareNextPartition (this=0x7f9218ca5400, state=0x3d11d100)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/exec/partitioned-hash-join-node.cc:685
#8  0x00007f9218ca5550 in ?? ()
#9  0x000000000094f21b in push_back (this=0x7f9218ca5400, min_size=<value optimized out>, check_limits=<value optimized out>)
    at /usr/include/c++/4.4.7/bits/stl_vector.h:741
#10 impala::MemPool::FindChunk (this=0x7f9218ca5400, min_size=<value optimized out>, check_limits=<value optimized out>)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/runtime/mem-pool.cc:140
#11 0x00007f93aaa56370 in ?? ()
#12 0x0000000000000000 in ?? ()
#0  0x0000003144e32625 in raise () from /lib64/libc.so.6
#1  0x0000003144e33e05 in abort () from /lib64/libc.so.6
#2  0x00007f984e5cda55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3  0x00007f984e74df87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4  0x00007f984e5d296f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x0000000000c15c2e in impala::PartitionedHashJoinNode::LargestSpilledPartition (this=0x7f9218ca5400)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/exec/partitioned-hash-join-node.cc:719
#7  0x0000000000c1f08b in impala::PartitionedHashJoinNode::PrepareNextPartition (this=0x7f9218ca5400, state=0x3d11d100)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/exec/partitioned-hash-join-node.cc:685
#8  0x00007f9218ca5550 in ?? ()
#9  0x000000000094f21b in push_back (this=0x7f9218ca5400, min_size=<value optimized out>, check_limits=<value optimized out>)
    at /usr/include/c++/4.4.7/bits/stl_vector.h:741
#10 impala::MemPool::FindChunk (this=0x7f9218ca5400, min_size=<value optimized out>, check_limits=<value optimized out>)
    at /usr/src/debug/impala-2.2.0-cdh5.4.2/be/src/runtime/mem-pool.cc:140
#11 0x00007f93aaa56370 in ?? ()
#12 0x0000000000000000 in ?? ()

Each hss_err indicate the same thing

# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [impalad+0x815c2e] impala::PartitionedHashJoinNode::LargestSpilledPartition() const+0x2e
#

The line of code:

int64_t PartitionedHashJoinNode::LargestSpilledPartition() const {
  int64_t max_rows = 0;
  for (int i = 0; i < hash_partitions_.size(); ++i) {
    Partition* partition = hash_partitions_[i];
    if (partition->is_spilled()) { <==========crash here, partition seems to be NULL
      int64_t rows = partition->build_rows()->num_rows();
      if (rows > max_rows) max_rows = rows;
    }
  }
  return max_rows;
}