[IMPALA-2285] Query should return error not cancelled when failing to flush blocks due to lack of disk space

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: Impala 2.1.2
• Component/s: Backend
• Labels: ramp-up resource-management
• Target Version: Product Backlog
• Assignee: Tim Armstrong
• Reporter: Matthew Jacobs
• Created: 02/Sep/15 14:43
• Updated: 20/Apr/17 20:08
• Resolved: 30/Mar/17 20:02

Description

When BufferedBlockMgr blocks must be flushed to disk but the write cannot be completed due to lack of disk space on the scratch disks, queries fail with a cancelled status and no further information. Queries should instead fail with an exception status and an error message indicating the scratch disk was full.

This can be reproduced by setting the scratch dir to a disk with negligible space and forcing a query to spill to disk. The way to determine this occurred is by looking for the following in the impalad log:

I0831 14:13:35.791662 14318 buffered-block-mgr.cc:769] Query: f74317d168f9d915:58d67871e44fa2a2 write complete callback with error.
I0831 14:13:35.791698 14318 runtime-state.cc:225] Error from query f74317d168f9d915:58d67871e44fa2a2: posix_fallocate(279, 17095983104, 8388527) failed for file /tmp/impala-scratch/f74317d168f9d915:58d67871e44fa2a2_747758b8-ff16-4a00-8d1f-0566ab71c787 with returnval=28 description=Error(11): Resource temporarily unavailable

There is no workaround.

Issue Links

• is blocked by: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
• is related to: IMPALA-5234 Get rid of redundant LogError() messages
• relates to: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool