[DRILL-5732] Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill. - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Not A Problem
Affects Version/s: 1.10.0
Fix Version/s: None
Component/s: None
Labels:
None

Description

git commit id:

| 1.12.0-SNAPSHOT  | e9065b55ea560e7f737d6fcb4948f9e945b9b14f  | DRILL-5660: Parquet metadata caching improvements  | 15.08.2017 @ 09:31:00 PDT  | root@qa-node190.qa.lab  | 15.08.2017 @ 13:29:26 PDT  |

Query is:

ALTER SESSION SET `exec.sort.disable_managed` = false;
alter session set `planner.disable_exchanges` = true;
alter session set `planner.memory.max_query_memory_per_node` = 104857600;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.width.max_per_query` = 1;
select max(col1), max(cs_sold_date_sk), max(cs_sold_time_sk), max(cs_ship_date_sk), max(cs_bill_customer_sk), max(cs_bill_cdemo_sk), max(cs_bill_hdemo_sk), max(cs_bill_addr_sk), max(cs_ship_customer_sk), max(cs_ship_cdemo_sk), max(cs_ship_hdemo_sk), max(cs_ship_addr_sk), max(cs_call_center_sk), max(cs_catalog_page_sk), max(cs_ship_mode_sk), min(cs_warehouse_sk), max(cs_item_sk), max(cs_promo_sk), max(cs_order_number), max(cs_quantity), max(cs_wholesale_cost), max(cs_list_price), max(cs_sales_price), max(cs_ext_discount_amt), min(cs_ext_sales_price), max(cs_ext_wholesale_cost), min(cs_ext_list_price), min(cs_ext_tax), min(cs_coupon_amt), max(cs_ext_ship_cost), max(cs_net_paid), max(cs_net_paid_inc_tax), min(cs_net_paid_inc_ship), min(cs_net_paid_inc_ship_tax), min(cs_net_profit), min(c_customer_sk), min(length(c_customer_id)), max(c_current_cdemo_sk), max(c_current_hdemo_sk), min(c_current_addr_sk), min(c_first_shipto_date_sk), min(c_first_sales_date_sk), min(length(c_salutation)), min(length(c_first_name)), min(length(c_last_name)), min(length(c_preferred_cust_flag)), max(c_birth_day), min(c_birth_month), min(c_birth_year), max(c_last_review_date), c_email_address  from (select cs_sold_date_sk+cs_sold_time_sk col1, * from dfs.`/drill/testdata/resource-manager/md1362` order by c_email_address nulls first) d where d.col1 > 2536816 and c_email_address is not null group by c_email_address;
ALTER SESSION SET `exec.sort.disable_managed` = true;
alter session set `planner.disable_exchanges` = false;
alter session set `planner.memory.max_query_memory_per_node` = 2147483648;
alter session set `planner.width.max_per_node` = 17;
alter session set `planner.width.max_per_query` = 1000;

Here is the stack trace:

2017-08-18 13:15:27,052 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG o.a.d.e.t.g.SingleBatchSorterGen27 - Took 6445 us to sort 9039 records
2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG o.a.d.e.p.i.xsort.ExternalSortBatch - Copier allocator current allocation 0
2017-08-18 13:15:27,420 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] DEBUG o.a.d.e.p.i.xsort.ExternalSortBatch - mergeAndSpill: starting total size in memory = 71964288
2017-08-18 13:15:27,421 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] INFO  o.a.d.e.p.i.xsort.ExternalSortBatch - User Error Occurred: One or more nodes ran out of memory while executing the query.
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

Unable to allocate sv2 for 9039 records, and not enough batchGroups to spill.
batchGroups.size 1
spilledBatchGroups.size 0
allocated memory 71964288
allocator limit 52428800

[Error Id: 7b248f12-2b31-4013-86b6-92e6c842db48 ]
        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550) ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.newSV2(ExternalSortBatch.java:637) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:379) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.test.generated.HashAggregatorGen24.doWork(HashAggTemplate.java:568) [na:na]
        at org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext(HashAggBatch.java:168) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:133) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:105) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_111]
        at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_111]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) [hadoop-common-2.7.0-mapr-1607.jar:na]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_111]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
2017-08-18 13:15:27,421 [2668b522-5833-8fd2-0b6d-e685197f0ae3:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2668b522-5833-8fd2-0b6d-e685197f0ae3:0:0: State change requested RUNNING --> FAILED

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

drillbit.log.83
24/Aug/17 06:31
15 kB
Robert Hou
26621eb2-daec-cef9-efed-5986e72a750a.sys.drill
24/Aug/17 06:31
19 kB
Robert Hou

Activity

People

Assignee:: Paul Rogers

Reporter:: Robert Hou

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 18/Aug/17 20:28

Updated:: 30/Aug/17 01:47

Resolved:: 30/Aug/17 01:47