[HDDS-1477] Recon Server stops after failed attempt to get snapshot from OM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.4.0
• Component/s: Ozone Recon
• Target Version/s: 0.5.0
• Assignee: Aravindan Vijayan
• Reporter: Vivek Ratnavel Subramanian
• Created: 26/Apr/19 23:40
• Updated: 29/Apr/19 19:55
• Resolved: 29/Apr/19 19:55

Description

Recon server stop after it is unable to connect to om

 

2019-04-26 14:55:03,441 INFO org.apache.hadoop.utils.db.DBStoreBuilder: using custom profile for table: default
2019-04-26 14:55:03,441 INFO org.apache.hadoop.utils.db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:default
2019-04-26 14:55:03,444 INFO org.apache.hadoop.utils.db.DBStoreBuilder: Using default options. DBProfile.DISK
2019-04-26 14:55:03,659 INFO org.apache.hadoop.conf.Configuration.deprecation: No unit for recon.om.connection.request.timeout(5000) assuming MILLISECONDS
2019-04-26 14:56:05,389 INFO org.apache.hadoop.ozone.recon.tasks.ContainerKeyMapperTask: Starting a run of ContainerKeyMapperTask.
2019-04-26 14:56:05,454 ERROR org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl: Unable to obtain Ozone Manager DB Snapshot.
org.apache.http.conn.HttpHostConnectException: Connect to 0.0.0.0:9874 [/0.0.0.0] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.hadoop.ozone.recon.ReconUtils.makeHttpCall(ReconUtils.java:161)
at org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.getOzoneManagerDBSnapshot(OzoneManagerServiceProviderImpl.java:173)
at org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl.updateReconOmDBWithNewSnapshot(OzoneManagerServiceProviderImpl.java:144)
at org.apache.hadoop.ozone.recon.tasks.ContainerKeyMapperTask.run(ContainerKeyMapperTask.java:69)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
... 20 more
2019-04-26 14:56:05,456 ERROR org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl: Null snapshot location got from OM.
2019-04-26 16:09:09,557 INFO org.apache.hadoop.ozone.recon.ReconServer: Stopping Recon server