[IMPALA-3105] RowBatch::MaxTupleBufferSize() calculation incorrect, may lead to memory corruption

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: crash
• Target Version: Impala 2.6.0
• Assignee: Tim Armstrong
• Reporter: casey
• Created: 01/Mar/16 19:15
• Updated: 15/Apr/16 01:08
• Resolved: 15/Apr/16 01:08

Description

An Impala crash was reported on the Kudu user list [1]. The problem seems to be that the calculation below is incorrect

5.7 code:

int RowBatch::MaxTupleBufferSize() {
  int row_size = row_desc_.GetRowSize();
  if (row_size > AT_CAPACITY_MEM_USAGE) return row_size;
  int num_rows = 0;
  if (row_size != 0) {
    num_rows = std::min(capacity_, AT_CAPACITY_MEM_USAGE / row_size);
  }
  int tuple_buffer_size = num_rows * row_size;
  DCHECK_LE(tuple_buffer_size, AT_CAPACITY_MEM_USAGE);
  return tuple_buffer_size;
}

Tim says AT_CAPACITY_MEM_USAGE should be considered a "soft limit" and exec nodes are allowed to use the full 'capacity_' if needed.

The query that exposes the problem is

SELECT * FROM <text_table_with_1000_cols> 
UNION ALL
SELECT * FROM <text_table_with_1000_cols>

If the lines

 
  if (row_size != 0) {
    num_rows = std::min(capacity_, AT_CAPACITY_MEM_USAGE / row_size);
  }

are change to

  if (row_size != 0) num_rows = capacity_;

then the query seems to run.

From talking to Tim, this isnt a good solution because something should be done to enforce a memory based limit on row batches.

NOTE: The problem was found using 0.6 Kimpala which is based on a commit between 2.2 and 2.3. Also the problem couldn't be reproduced on current trunk (just before 2.5).

1: http://mail-archives.apache.org/mod_mbox/incubator-kudu-user/201602.mbox/browser

Issue Links

• relates to: IMPALA-3069 mem-pool.cc:254] Check failed: chunks_[i].allocated_bytes > 0 (-2104452104 vs. 0)