[DRILL-4542] if external sort fails to spill to disk, memory is leaked and wrong error message is displayed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.0
• Fix Version/s: 1.9.0
• Component/s: Execution - Relational Operators

Environment

When external sort fails spilling to disk, for example because there is no space left on disk, the query fails with an error like this:

SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (376832)
Allocator(op:1:0:6:ExternalSort) 20000000/376832/408432256/429496729 (res/actual/peak/limit)

2 problems here:

1. we are leaking memory

2. the "real" problem is not shown in the error message

Looking at the logs we can find the real issue, generally right after the previous error (edited for the sake of clarity):

2016-03-26 02:22:28,576 [drill-executor-119] ERROR o.a.d.exec.server.BootStrapContext - org.apache.drill.exec.work.WorkManager$WorkerBee$2.run() leaked an exception.
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:249) ~[hadoop-common-2.7.0-mapr-1602.jar:na]
		[...]
        at org.apache.drill.exec.cache.VectorAccessibleSerializable.writeToStream(VectorAccessibleSerializable.java:172) ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.xsort.BatchGroup.addBatch(BatchGroup.java:96) ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill(ExternalSortBatch.java:562) ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:394) ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
		[...]

• Assignee: Boaz Ben-Zvi
• Reporter: Abdel Hakim Deneche
• Created: 26/Mar/16 09:53
• Updated: 22/Sep/16 20:51
• Resolved: 22/Sep/16 20:51

Issue Links

• is duplicated by: DRILL-3898 No space error during external sort does not cancel the query
• is duplicated by: DRILL-4616 Running out of /tmp results in "Memory was leaked by query"