[FLINK-2506] HBase connection closing down (table distributed over more than 1 region server - Flink Cluster-Mode)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.10.0
• Component/s: Examples
• Assignee: Aljoscha Krettek
• Reporter: Lydia Ickler
• Created: 11/Aug/15 06:22
• Updated: 13/Oct/17 16:04
• Resolved: 13/Oct/17 16:04

Description

If I fill a default table (create 'test-table', 'someCf') with the HBaseWriteExample.java program from the HBase addon library then a table without start/end key is created.
The data reading works great with the HBaseReadExample.java.

Nevertheless, if I manually create a "test-table" that is distributed over more than one region server (create 'test-table2', 'someCf',

{NUMREGIONS => 3,SPLITALGO => 'HexStringSplit'}

) the run is canceled with the following error message:

grips2
Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Fri Aug 07 11:18:29 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:18:38 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:18:48 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:18:58 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:08 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:18 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:28 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:38 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:48 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:19:58 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:20:18 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:20:38 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:20:58 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:21:19 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:21:39 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:21:59 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:22:19 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:22:39 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:22:59 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:23:19 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:23:39 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:24:00 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:24:20 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:24:40 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:25:00 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:25:20 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:25:40 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:26:00 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:26:20 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:26:40 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:27:00 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:27:20 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:27:40 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:28:01 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed
Fri Aug 07 11:28:21 CEST 2015, org.apache.hadoop.hbase.client.RpcRetryingCaller@28961f68, java.io.IOException: hconnection-0x47bf79d7 closed

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:131)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:284)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:189)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:184)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:110)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:775)
at org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:153)
at org.apache.flink.addons.hbase.TableInputFormat.open(TableInputFormat.java:48)
at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:151)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: hconnection-0x47bf79d7 closed
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(HConnectionManager.java:1812)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.isTableOnlineState(ZooKeeperRegistry.java:100)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.isTableDisabled(HConnectionManager.java:980)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:1133)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:958)
at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:72)
at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:125)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
... 11 more