[IMPALA-11990] Output on Ozone test run failure without snapshot is confusing

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Epic Color: ghx-label-4
• Assignee: Michael Smith
• Reporter: Gabor Kaszab
• Created: 09/Mar/23 14:50
• Updated: 12/Mar/23 23:55
• Resolved: 10/Mar/23 22:23

Description

This happens during setting up the minicluster after a successful Ozone build:

16:55:30 23/03/07 07:55:34 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy10.submitRequest over nodeId=null,nodeAddress=localhost:9862 after 1 failover attempts. Trying to failover after sleeping for 4000ms.
16:55:34 23/03/07 07:55:38 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy10.submitRequest over nodeId=null,nodeAddress=localhost:9862 after 2 failover attempts. Trying to failover after sleeping for 6000ms.
16:55:38 23/03/07 07:55:44 ERROR ha.OMFailoverProxyProviderBase: Failed to connect to OMs: [om1]. Attempted 3 failovers.
16:55:44 23/03/07 07:55:44 ERROR client.OzoneClientFactory: Couldn't create RpcClient protocol exception: 
16:55:44 java.net.ConnectException: Call From localhost/127.0.0.1 to localhost:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
16:55:44 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
16:55:44 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
16:55:44 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
16:55:44 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
16:55:44 	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:892)
16:55:44 	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:812)
16:55:44 	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1617)
16:55:44 	at org.apache.hadoop.ipc.Client.call(Client.java:1559)
16:55:44 	at org.apache.hadoop.ipc.Client.call(Client.java:1456)
16:55:44 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
16:55:44 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
16:55:44 	at com.sun.proxy.$Proxy10.submitRequest(Unknown Source)
16:55:44 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:55:44 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:55:44 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:55:44 	at java.lang.reflect.Method.invoke(Method.java:498)
16:55:44 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
16:55:44 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
16:55:44 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
16:55:44 	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
16:55:44 	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
16:55:44 	at com.sun.proxy.$Proxy10.submitRequest(Unknown Source)
16:55:44 	at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
16:55:44 	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
16:55:44 	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
16:55:44 	at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:242)
16:55:44 	at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
16:55:44 	at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:230)
16:55:44 	at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:93)
16:55:44 	at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.<init>(BasicRootedOzoneClientAdapterImpl.java:186)
16:55:44 	at org.apache.hadoop.fs.ozone.RootedOzoneClientAdapterImpl.<init>(RootedOzoneClientAdapterImpl.java:51)
16:55:44 	at org.apache.hadoop.fs.ozone.RootedOzoneFileSystem.createAdapter(RootedOzoneFileSystem.java:93)
16:55:44 	at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.initialize(BasicRootedOzoneFileSystem.java:163)
16:55:44 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452)
16:55:44 	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162)
16:55:44 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557)
16:55:44 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504)
16:55:44 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522)
16:55:44 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:270)
16:55:44 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:506)
16:55:44 	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
16:55:44 	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:347)
16:55:44 	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:252)
16:55:44 	at org.apache.hadoop.fs.shell.Delete$Rm.expandArgument(Delete.java:93)
16:55:44 	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:235)
16:55:44 	at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
16:55:44 	at org.apache.hadoop.fs.shell.Command.run(Command.java:179)
16:55:44 	at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
16:55:44 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
16:55:44 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
16:55:44 	at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
16:55:44 Caused by: java.net.ConnectException: Connection refused
16:55:44 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
16:55:44 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
16:55:44 	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:205)
16:55:44 	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
16:55:44 	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:727)
16:55:44 	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:840)
16:55:44 	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:430)
16:55:44 	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1678)
16:55:44 	at org.apache.hadoop.ipc.Client.call(Client.java:1503)
16:55:44 	... 43 more
16:55:44 23/03/07 07:55:44 WARN fs.FileSystem: Failed to initialize fileystem ofs://localhost:9862: java.net.ConnectException: Connection refused
16:55:44 rm: Connection refused