[DRILL-4708] connection closed unexpectedly

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.7.0
• Fix Version/s: 1.13.0
• Component/s: Execution - RPC
• Assignee: Vlad Rozov
• Reporter: Chun Chang
• Created: 03/Jun/16 23:10
• Updated: 28/Feb/18 00:38
• Resolved: 28/Feb/18 00:38

Description

Running DRILL functional automation, we often see query failed randomly due to the following unexpected connection close error.

Execution Failures:
/root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/100000rows/filter5.q
Query: 
select * from dfs.ctas_flatten.`filter5_100000rows_ctas`
Failed with exception
java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. Drillbit down?


[Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
	at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
	at oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
	at org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
	at org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. Drillbit down?


[Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
	at oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
	at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
	at oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
	at oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
	at oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
	at oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
	at oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
	at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	... 1 more