Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20771

PUT operation fail with "No server address listed in hbase:meta for region xxxxx"

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.3.3, 1.4.6, 1.2.7
    • Component/s: Client
    • Labels:
      None

      Description

      1) Create a table with 1 region

      2) AM send RPC to RS to open the region,

       // invoke assignment (async)
       ArrayList<HRegionInfo> userRegionSet = new ArrayList<HRegionInfo>(regions);
       for (Map.Entry<ServerName, List<HRegionInfo>> plan: bulkPlan.entrySet()) {
       if (!assign(plan.getKey(), plan.getValue())) {
       for (HRegionInfo region: plan.getValue()) {
       if (!regionStates.isRegionOnline(region)) {
       invokeAssign(region);
       if (!region.getTable().isSystemTable()) {
       userRegionSet.add(region);
       }
       }
       }
       }
       }
      

      In above code if assignment fails (due to some problem) then AM will sumbit the assign request to the thread pool and wait for some duration (here it will wait for 20sec, region count is 1).

      3) After 20sec, CreateTableProcedure will set table state as ENABLED in ZK and finish the procedure.

       // Mark the table as Enabling
       assignmentManager.getTableStateManager().setTableState(tableName,
       ZooKeeperProtos.Table.State.ENABLING);
      
      // Trigger immediate assignment of the regions in round-robin fashion
       ModifyRegionUtils.assignRegions(assignmentManager, regions);
      
      // Enable table
       assignmentManager.getTableStateManager()
       .setTableState(tableName, ZooKeeperProtos.Table.State.ENABLED);
      

      4) At the client side, CreateTableFuture.waitProcedureResult(...) is waiting for the procedure to finish,

       // If the procedure is no longer running, we should have a result
       if (response != null && response.getState() != GetProcedureResultResponse.State.RUNNING) {
       procResultFound = response.getState() != GetProcedureResultResponse.State.NOT_FOUND;
       return convertResult(response);
       }
      

      Here we wait for operation result only when procedure result not found, but in this scenario it will be wrong because region assignment didnt complete,

       // if we don't have a proc result, try the compatibility wait
       if (!procResultFound) {
       result = waitOperationResult(deadlineTs);
       }
      

      Since HBaseAdmin didn't wait for operation result (successful region assignment), so client PUT operation will fail by the time region is successfully opened because "info:server" entry wont be there in meta.

        Attachments

        1. HBASE-20771.branch-1.002.patch
          5 kB
          Pankaj Kumar
        2. HBASE-20771.branch-1.001.patch
          5 kB
          Pankaj Kumar

          Activity

            People

            • Assignee:
              pankaj2461 Pankaj Kumar
              Reporter:
              pankaj2461 Pankaj Kumar

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment