[HBASE-20771] PUT operation fail with "No server address listed in hbase:meta for region xxxxx"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.3.3, 1.4.6, 1.2.7
• Component/s: Client
• Assignee: Pankaj Kumar
• Reporter: Pankaj Kumar
• Created: 21/Jun/18 16:41
• Updated: 01/Feb/19 20:08
• Resolved: 09/Jul/18 23:08

Description

1) Create a table with 1 region

2) AM send RPC to RS to open the region,

 // invoke assignment (async)
 ArrayList<HRegionInfo> userRegionSet = new ArrayList<HRegionInfo>(regions);
 for (Map.Entry<ServerName, List<HRegionInfo>> plan: bulkPlan.entrySet()) {
 if (!assign(plan.getKey(), plan.getValue())) {
 for (HRegionInfo region: plan.getValue()) {
 if (!regionStates.isRegionOnline(region)) {
 invokeAssign(region);
 if (!region.getTable().isSystemTable()) {
 userRegionSet.add(region);
 }
 }
 }
 }
 }

In above code if assignment fails (due to some problem) then AM will sumbit the assign request to the thread pool and wait for some duration (here it will wait for 20sec, region count is 1).

3) After 20sec, CreateTableProcedure will set table state as ENABLED in ZK and finish the procedure.

 // Mark the table as Enabling
 assignmentManager.getTableStateManager().setTableState(tableName,
 ZooKeeperProtos.Table.State.ENABLING);

// Trigger immediate assignment of the regions in round-robin fashion
 ModifyRegionUtils.assignRegions(assignmentManager, regions);

// Enable table
 assignmentManager.getTableStateManager()
 .setTableState(tableName, ZooKeeperProtos.Table.State.ENABLED);

4) At the client side, CreateTableFuture.waitProcedureResult(...) is waiting for the procedure to finish,

 // If the procedure is no longer running, we should have a result
 if (response != null && response.getState() != GetProcedureResultResponse.State.RUNNING) {
 procResultFound = response.getState() != GetProcedureResultResponse.State.NOT_FOUND;
 return convertResult(response);
 }

Here we wait for operation result only when procedure result not found, but in this scenario it will be wrong because region assignment didnt complete,

 // if we don't have a proc result, try the compatibility wait
 if (!procResultFound) {
 result = waitOperationResult(deadlineTs);
 }

Since HBaseAdmin didn't wait for operation result (successful region assignment), so client PUT operation will fail by the time region is successfully opened because "info:server" entry wont be there in meta.