[PHOENIX-1735] Insert data in hbase through phoenix JDBC when restart hbase service

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 4.2.0
• Labels: Region assigned: become for in it still to transition, util.HBaseFsckRepair: waiting

Environment

linux os ( 64G ram,1.5T disk,16 cores) * 4
Hadoop 2.6.0
HBase 0.98.6
Phoenix 4.2.0

• Epic Link: Update Ambari OS Support
• Assignee: Unassigned
• Reporter: wubiandelvbo
• Created: 14/Mar/15 02:37
• Updated: 24/Jan/16 01:44
• Resolved: 24/Jan/16 01:44

Description

use four clients put data in hbase through phoenix-JDBC when restart hbase service。I discover hbase break down.
use:
hbase hbck

ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table CARD_INDEX_SALT
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.CATALOG
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TZJT_CAMERA_SALT
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table T
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:11:59,933 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TT
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TT3
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,026 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table hbase:namespace
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table TABLENAME
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.SEQUENCE
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table SYSTEM.STATS
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
2015-03-14 10:12:00,027 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: Found inconsistency in table LOCATION_INDEX_SALT
2015-03-14 10:12:01,248 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop1:2181,hadoop2:2181,hadoop3:2181 sessionTimeout=30000 watcher=hbase Fsck, quorum=hadoop1:2181,hadoop2:2181,hadoop3:2181, baseZNode=/hbase-unsecure
2015-03-14 10:12:01,248 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hadoop1:2181,hadoop2:2181,hadoop3:2181
2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Opening socket connection to server hadoop1/10.165.97.7:2181. Will not attempt to authenticate using SASL (unknown error)
2015-03-14 10:12:01,249 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Socket connection established to hadoop1/10.165.97.7:2181, initiating session
2015-03-14 10:12:01,252 INFO [main-SendThread(hadoop1:2181)] zookeeper.ClientCnxn: Session establishment complete on server hadoop1/10.165.97.7:2181, sessionid = 0x14c1337679e00f9, negotiated timeout = 30000
Summary:
IDX_CC is okay.
Number of regions: 0
Deployed on:
CARD_INDEX is okay.
Number of regions: 0
Deployed on:
US_POPULATION is okay.
Number of regions: 1
Deployed on: hadoop4,60020,1426296975241
EXAMPLE2 is okay.
Number of regions: 1
Deployed on: hadoop2,60020,1426296977835
Table SKIPDEMO is inconsistent.
Number of regions: 9
Deployed on: hadoop3,60020,1426296976617 hadoop4,60020,1426296975241
STUDENT is okay.
Number of regions: 0
Deployed on:
LOCATION_INDEX is okay.
Number of regions: 0
Deployed on:
TZJT_CAMERA is okay.
Number of regions: 0
Deployed on:
hbase:meta is okay.
Number of regions: 1
Deployed on: hadoop2,60020,1426296977835
CARD_INDEX_SALT is okay.
Number of regions: 0
Deployed on:
SYSTEM.CATALOG is okay.
Number of regions: 0
Deployed on:
TZJT_CAMERA_SALT is okay.
Number of regions: 0
Deployed on:
T is okay.
Number of regions: 0
Deployed on:
TT is okay.
Number of regions: 0
Deployed on:
TT3 is okay.
Number of regions: 0
Deployed on:
hbase:namespace is okay.
Number of regions: 0
Deployed on:
TABLENAME is okay.
Number of regions: 0
Deployed on:
SYSTEM.SEQUENCE is okay.
Number of regions: 0
Deployed on:
SYSTEM.STATS is okay.
Number of regions: 0
Deployed on:
LOCATION_INDEX_SALT is okay.
Number of regions: 0
Deployed on:
462 inconsistencies detected.
Status: INCONSISTENT
2015-03-14 10:12:01,255 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService

so,use 'hbase hbck -fixMeta -fixAssignments hbase:namespace '，'hbase hbck -repair hbase:namespace' show:

2015-03-14 10:18:14,988 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

2015-03-14 10:18:15,994 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

2015-03-14 10:18:17,000 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

2015-03-14 10:18:18,005 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

2015-03-14 10:18:19,011 INFO [main] util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned:

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2015-03-14 10:18:24,041 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24c15ce67b80047
2015-03-14 10:18:24,045 INFO [main] zookeeper.ZooKeeper: Session: 0x24c15ce67b80047 closed
2015-03-14 10:18:24,045 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-03-14 10:18:24,045 DEBUG [main] ipc.RpcClient: Stopping rpc client
Exception in thread "main" java.io.IOException: Region

{ENCODED => 7670aca9fd6d0709f2df8d0c6c81dc10, NAME => 'hbase:namespace,,1425450589482.7670aca9fd6d0709f2df8d0c6c81dc10.', STARTKEY => '', ENDKEY => ''}

failed to move out of transition within timeout 120000ms
at org.apache.hadoop.hbase.util.HBaseFsckRepair.waitUntilAssigned(HBaseFsckRepair.java:141)
at org.apache.hadoop.hbase.util.HBaseFsck.tryAssignmentRepair(HBaseFsck.java:1798)
at org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:1969)
at org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1579)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:473)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:492)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4213)
at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4022)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4010)

------------------------------
the hbase still not normal.
I don't know why ?
Is bug ?
My operational problems?
Can someone help me
Thanks!