Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
-
None
Description
SwitchToIoBuffers sets the use_small_buffers_ flag of the stream to false and calls NewBlockForWrite(). But if NewBlockForWrite() fails to get a new block, that is, in case block_mgr_->GetNewBlock() fails, then got_buffer is false but at the same time use_small_buffers_ has now switched to false.
That means that any subsequent attempt to recover, for example by spilling a different partition and trying to append to this stream will fail, because this time around use_small_buffers_ is wrongly false.
Status BufferedTupleStream::SwitchToIoBuffers(bool* got_buffer) {
if (!use_small_buffers_) {
*got_buffer = (write_block_ != NULL);
return Status::OK();
}
use_small_buffers_ = false;
return NewBlockForWrite(block_mgr_->max_block_size(), got_buffer);
}
In NewBlockForWrite():
BufferedBlockMgr::Block* new_block = NULL;
{
SCOPED_TIMER(get_new_block_timer_);
RETURN_IF_ERROR(block_mgr_->GetNewBlock(
block_mgr_client_, unpin_block, &new_block, block_len)); <== if GetNewBlock() does not get a new block because of memory pressure, SwitchToIoBuffers will return leaving using_small_buffers_ to false.
}
*got_block = (new_block != NULL);
if (!*got_block) {
DCHECK(unpin_block == NULL);
return Status::OK();
}
It seems that we should simply setting using_small_buffers_ back to true in case got_buffer==false;