Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
-
None
Description
SwitchToIoBuffers sets the use_small_buffers_ flag of the stream to false and calls NewBlockForWrite(). But if NewBlockForWrite() fails to get a new block, that is, in case block_mgr_->GetNewBlock() fails, then got_buffer is false but at the same time use_small_buffers_ has now switched to false.
That means that any subsequent attempt to recover, for example by spilling a different partition and trying to append to this stream will fail, because this time around use_small_buffers_ is wrongly false.
Status BufferedTupleStream::SwitchToIoBuffers(bool* got_buffer) { if (!use_small_buffers_) { *got_buffer = (write_block_ != NULL); return Status::OK(); } use_small_buffers_ = false; return NewBlockForWrite(block_mgr_->max_block_size(), got_buffer); } In NewBlockForWrite(): BufferedBlockMgr::Block* new_block = NULL; { SCOPED_TIMER(get_new_block_timer_); RETURN_IF_ERROR(block_mgr_->GetNewBlock( block_mgr_client_, unpin_block, &new_block, block_len)); <== if GetNewBlock() does not get a new block because of memory pressure, SwitchToIoBuffers will return leaving using_small_buffers_ to false. } *got_block = (new_block != NULL); if (!*got_block) { DCHECK(unpin_block == NULL); return Status::OK(); }
It seems that we should simply setting using_small_buffers_ back to true in case got_buffer==false;