[DRILL-5740] hash agg fail to read spill file

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: Functions - Drill
• Assignee: Boaz Ben-Zvi
• Reporter: Chun Chang
• Reviewer: Jinfeng Ni
• Created: 25/Aug/17 18:51
• Updated: 25/Sep/17 21:34
• Resolved: 20/Sep/17 23:09

Description

-Build: | 1.12.0-SNAPSHOT | 11008d029bafa36279e3045c4ed1a64366080620
-Multi-node drill cluster

Running a query causing hash agg spill fails with the following error. And this seems to be a regression.

Execution Failures:
/root/drill-test-framework/framework/resources/Advanced/hash-agg/spill/hagg5.q
Query:
select gby_date, gby_int32_rand, sum(int32_field), avg(float_field), min(boolean_field), count(double_rand) from dfs.`/drill/testdata/hagg/PARQUET-500M.parquet` group by gby_date, gby_int32_rand order by gby_date, gby_int32_rand limit 30
Failed with exception
java.sql.SQLException: SYSTEM ERROR: FileNotFoundException: File /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3 does not exist

Fragment 1:34

[Error Id: 291a79f8-9b7a-485d-9404-e7b7fe1d8f1e on 10.10.30.168:31010]

  (java.lang.RuntimeException) java.io.FileNotFoundException: File /tmp/drill/spill/10.10.30.168-31010/265f91f9-78d2-78a6-68ad-4709674efe0a_HashAgg_1-4-34/spill3 does not exist
    org.apache.drill.exec.physical.impl.aggregate.SpilledRecordbatch.<init>():67
    org.apache.drill.exec.test.generated.HashAggregatorGen1891.outputCurrentBatch():980
    org.apache.drill.exec.test.generated.HashAggregatorGen1891.doWork():617
    org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():168
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():133
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.physical.impl.TopN.TopNBatch.innerNext():191
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.record.AbstractRecordBatch.next():119
    org.apache.drill.exec.record.AbstractRecordBatch.next():109
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
    org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext():93
    org.apache.drill.exec.record.AbstractRecordBatch.next():164
    org.apache.drill.exec.physical.impl.BaseRootExec.next():105
    org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
    org.apache.drill.exec.physical.impl.BaseRootExec.next():95
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():415
    org.apache.hadoop.security.UserGroupInformation.doAs():1595
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1145
    java.util.concurrent.ThreadPoolExecutor$Worker.run():615
    java.lang.Thread.run():745

Issue Links

• incorporates: DRILL-5694 hash agg spill to disk, second phase OOM
• is part of: DRILL-5694 hash agg spill to disk, second phase OOM
• links to: GitHub Pull Request #924