[AMBARI-25362] Hive View throws TimeoutException deadline passed for few queries randomly.

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.2
• Component/s: ambari-views
• Labels: pull-request-available
• Assignee: Sreenath Somarajapuram
• Reporter: Jay SenSharma
• Created: 14/Aug/19 23:19
• Updated: 01/Oct/19 06:43

Description

Running some hive queries throws "deadline passed" error as following inside amber logs.

25 Jul 2019 13:07:53,134 ERROR [ambari-client-thread-591197] [HIVE 2.0.0 AUTO_HIVE20_INSTANCE] ServiceFormattedException:97 - deadline passed
25 Jul 2019 13:07:53,135 ERROR [ambari-client-thread-591197] [HIVE 2.0.0 AUTO_HIVE20_INSTANCE] ServiceFormattedException:98 - java.util.concurrent.TimeoutException: deadline passed

java.util.concurrent.TimeoutException: deadline passed
	at akka.actor.dsl.Inbox$InboxActor$$anonfun$receive$1.applyOrElse(Inbox.scala:117)
	at scala.PartialFunction$AndThen.applyOrElse(PartialFunction.scala:189)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
	at akka.actor.dsl.Inbox$InboxActor.aroundReceive(Inbox.scala:62)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Increasing the following timeouts inside "/etc/ambari-server/conf/ambari.properties" followed by ambari server restart does not help either.

views.ambari.hive.AUTO_HIVE20_INSTANCE.connection.inactivity.timeout=7200000
views.ambari.hive.AUTO_HIVE20_INSTANCE.result.fetch.timeout=7200000
views.ambari.request.connect.timeout.millis=7200000
views.ambari.request.read.timeout.millis=7200000
views.request.connect.timeout.millis=7200000
views.request.read.timeout.millis=7200000

Additionally setting the following property in HiveView20 and the executing the query still causes the same failure.

set hive.fetch.task.conversion=none  

"TimeoutException: deadline passed" still occurs much prior to the timeout defined.

Issue Links

• links to: GitHub Pull Request #3072