Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Below is the summary of issue:
Scenario:
It seems like first sendRecordBatch was sent to Foreman which initiated the Authentication handshake. But before initiating handshake for auth we establish a connection and store that in a registry. Now if in parallel there is another recordBatch (by a different minor fragment running on same Drillbit) to be sent then that will see the connection available in registry and will initiate the send. Before the authentication is completed this second request reached foreman and it throws below exception saying RPC type 3 message is not allowed and closes the connection. This also fails the authentication handshake which was in progress. Here the logs with details:
Forman received the SASL_START message from another node:
2018-02-21 18:43:30,759 [BitServer-4] TRACE o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START from /10.10.100.161:35482
Then around same time it received another message from client of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not completed yet.
2018-02-21 18:43:30,762 [BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication. Connection: /10.10.100.162:31012 <--> /10.10.100.161:35482 (data server). Closing connection.
io.netty.handler.codec.DecoderException: org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed without authentication. Client on /10.10.100.161:35482 must authenticate before making requests. Connection dropped. [Details: Encryption: enabled , MaxWrappedSize: 65536 , WrapSizeLimit: 0]
Then client receives an channel closed exception:
2018-02-21 18:43:30,764 [BitClient-4] WARN o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed channel. Connection: /10.10.100.161:35482 <--> 10.10.100.162:31012 (data client)
and due to this it's initial command for authentication also fails. Since there is channel closed exception above I will think that triggered the failure of authentication request as well.
Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection. Failure type AUTHENTICATION.
at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83) ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.rpc.data.DataTunnel.sendRecordBatch(DataTunnel.java:84) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.ops.AccountingDataTunnel.sendRecordBatch(AccountingDataTunnel.java:45) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext(SingleSenderCreator.java:127) ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]
So I think there is a concurrency issue where even though the authentication is not completed the other requests are send to remote node as soon as TCP connection is available. Instead it should wait until authentication is completed. Something like TCP connection should be made available from registry only if authentication is completed.
Attachments
Issue Links
- links to
