[HDDS-2914] Certain Hive queries started to fail on generating splits

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Labels: pull-request-available
• Assignee: István Fajth
• Reporter: István Fajth
• Created: 20/Jan/20 10:31
• Updated: 20/Feb/20 23:20
• Resolved: 20/Feb/20 23:20

Description

After updating a cluster where I am running TPCDS queries, some queries started to fail.
The update happened from an early dec state to the jan 10 state of master.

Most likely the addition of HDDS-2188 is related to the problem, but it is still under investigation.

The exception I see in the queries:

[ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: inventory initializer failed, vertex=vertex_XXXX [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:328)
	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:817)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:753)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:634)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:110)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: ORC split generation failed with exception: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1915)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2002)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:532)
	at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:789)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
	... 5 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1909)
	... 17 more
Caused by: java.io.IOException: File o3fs://hive.warehouse.fqdn:9862/warehouse/tablespace/managed/hive/100/inventory/delta_0000001_0000001_0000/bucket_00000 should have had overlap on block starting at 0
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.createSplit(OrcInputFormat.java:1506)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateOrUpdateSplit(OrcInputFormat.java:1678)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.generateSplitsFromStripes(OrcInputFormat.java:1661)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1609)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2900(OrcInputFormat.java:1383)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1568)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator$1.run(OrcInputFormat.java:1565)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1565)
	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.call(OrcInputFormat.java:1383)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more

I haven't found any more specific stack traces so far.

Issue Links

• relates to: HDDS-2188 Implement LocatedFileStatus & getFileBlockLocations to provide node/localization information to Yarn/Mapreduce
• links to: GitHub Pull Request #540
• links to: GitHub Pull Request #549
• links to: GitHub Pull Request #563
• links to: GitHub Pull Request #563