[DRILL-5903] Regression: Query encounters "Waited for 15000ms, but tasks for 'Fetch parquet metadata' are not complete."

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Invalid
• Affects Version/s: 1.11.0
• Component/s: Metadata, Storage - Parquet
• Assignee: Arina Ielchiieva
• Reporter: Robert Hou
• Created: 24/Oct/17 06:37
• Updated: 21/May/19 13:12
• Resolved: 21/May/19 13:12

Description

This is a query from the Functional-Baseline-100.171 run. The test is /root/drillAutomation/mapr/framework/resources/Functional/parquet_storage/parquet_date/mc_parquet_date/generic/mixed1_partitioned5.q.

Query is:

select a.int_col, b.date_col from dfs.`/drill/testdata/parquet_date/metadata_cache/mixed/fewtypes_null_large` a inner join ( select date_col, int_col from dfs.`/drill/testdata/parquet_date/metadata_cache/mixed/fewtypes_null_large` where dir0 = '1.2' and date_col > '1996-03-07' ) b on cast(a.date_col as date)= date_add(b.date_col, 5) where a.int_col = 7 and a.dir0='1.9' group by a.int_col, b.date_col

From drillbit.log:

fc65-d430-ac1103638113: SELECT SUM(col_int) OVER() sum_int FROM vwOnParq_wCst_35
2017-10-23 11:20:50,122 [26122f83-6956-5aa8-d8de-d4808f572160:foreman] ERROR o.a.d.exec.store.parquet.Metadata - Waited for 15000ms, but tasks for 'Fetch parquet metadata' are not complete. Total runnable size 3, parallelism 3.
2017-10-23 11:20:50,127 [26122f83-6956-5aa8-d8de-d4808f572160:foreman] INFO  o.a.d.exec.store.parquet.Metadata - User Error Occurred: Waited for 15000ms, but tasks for 'Fetch parquet metadata' are not complete. Total runnable size 3, parallelism 3.
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Waited for 15000ms, but tasks for 'Fetch parquet metadata' are not complete. Total runnable size 3, parallelism 3.


[Error Id: 7484e127-ea41-4797-83c0-6619ea9b2bcd ]
	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) ~[drill-common-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.TimedRunnable.run(TimedRunnable.java:151) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.Metadata.getParquetFileMetadata_v3(Metadata.java:341) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:318) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.Metadata.getParquetTableMetadata(Metadata.java:142) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.ParquetGroupScan.init(ParquetGroupScan.java:934) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.ParquetGroupScan.<init>(ParquetGroupScan.java:227) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.ParquetGroupScan.<init>(ParquetGroupScan.java:190) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:170) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan(ParquetFormatPlugin.java:66) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan(FileSystemPlugin.java:144) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:100) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:85) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch(DrillPushProjIntoScan.java:62) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228) [calcite-core-1.4.0-drill-r22.jar:1.4.0-drill-r22]
	at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:811) [calcite-core-1.4.0-drill-r22.jar:1.4.0-drill-r22]
	at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:310) [calcite-core-1.4.0-drill-r22.jar:1.4.0-drill-r22]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:400) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:342) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:241) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:291) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:168) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:131) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:991) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:276) [drill-java-exec-1.12.0-SNAPSHOT.jar:1.12.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_141]
	at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141]
[root@qa-node209 log.nightlyrun.parquet]# 

sys.version is:
1.12.0-SNAPSHOT f1d1945b3772bb782039fd6811e34a7de66441c8 DRILL-5582: C++ Client: [Threat Modeling] Drillbit may be spoofed by an attacker and this may lead to data being written to the attacker's target instead of Drillbit 19.10.2017 @ 17:13:05 PDT Unknown 19.10.2017 @ 18:37:45 PDT

query ID is 26122f83-6956-5aa8-d8de-d4808f572160

Issue Links

• is related to: DRILL-5908 Regression: Query intermittently may fail with error "Waited for 15000ms, but tasks for 'Get block maps' are not complete."