[HBASE-21512] Reimplement sync client based on async client

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1
• Component/s: asyncclient, Client
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Reimplement sync client based on async client, the old code for sync client have been purged.

Most public APIs are kept, except:

1. coprocessor related methods in sync client interface have been marked as deprecated. This is because that these methods may reference methods from the BlockingInterface of a protobuf stub, so we have to use a thread pool to execute these methods which are not good. Please use the coprocessor methods in async client interface instead.

2. BufferedMutatorParams.setWriteBufferPeriodicFlushTimerTickMs has been marked as deprecated. It has no effect now, just kept for compatibility. We use a global timer per connection so you can not customize the tick any more.

3. BufferedMutatorParams.implementationClassName has been marked as deprecated. It has no effect now, as we will pass several internal stuff to the constructor when creating the implementation instance, which is not good. We should find another way to do it.

4. The behavior for the two Admin.split methods is changed a bit. Now when the methods return, the split should have been done, while in the past it just means we have send the request out.

5. Now most(non-deprecated) methods in Admin and Table interface do not accept null values any more.

6. LoadIncrementalHFiles has been removed. It was marked as deprecated in 2.2.0 so by default is should only be removed in 4.0.0. But it is an implementation class, neither an interface nor a data structure, it is not a good practice to make it public as it exposes several internal states, and is also easily to be misused in the future, so we just remove it in 3.0.0. Users can use BulkLoadHFiles in your code, and there is no difference when executing bulk load from command line.

And we also introduce a toAsyncConnection method in the Connection interface, and also a toConnection method in the AsyncConnection interface. They are both lightweighted operations, the returned instance will share the same connection to the hbase cluster, so you can easily convert Connection to AsyncConnection and vice visa.

For internal developers, the ClusterConnection has also been purged. Please use the new AsyncClusterConnection instead, and please also see the AsyncRegionServerAdmin, which wraps the call to a region server. There is no sync interface any more, please use FutureUtils.get on the returned Future if you want the sync behavior. And also the ShortCircultConnection has been removed, as now we will use async stub but our RpcServices only implement BlockingInterface.

• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 25/Nov/18 11:42
• Updated: 15/May/23 15:17
• Resolved: 18/Jun/19 13:58

Description

At least for the RSProcedureDispatcher, with CompletableFuture we do not need to set a delay and use a thread pool any more, which could reduce the resource usage and also the latency.

Once this is done, I think we can remove the ClusterConnection completely, and start to rewrite the old sync client based on the async client, which could reduce the code base a lot for our client.

Issue Links

• causes: HBASE-24890 The command regioninfo is not available
• duplicates: HBASE-18334 Remove sync client implementation and wrap async client under sync client interface
• is blocked by: HBASE-22590 Remove the deprecated methods in Table interface
• relates to: HBASE-22589 Change our RpcServices to implement protobuf Interface instead of BlockingInterface
• links to: GitHub Pull Request #287

Sub-Tasks

• Also initialize an AsyncClusterConnection in HRegionServer: Resolved
• Use AsyncConnection instead of Connection in SecureBulkLoadManager: Resolved
• Use AsyncClusterConnection in ServerManager for getRsAdmin: Resolved
• Rewrite ServerManager.closeRegionSilentlyAndWait to use AsyncClusterConnection: Resolved
• Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection: Resolved
• Use AsyncClusterConnection for HBaseInterClusterReplicationEndpoint: Resolved
• Remove ClusterConnection: Resolved
• Rewrite RegionReplicaReplicationEndpoint to use AsyncClusterConnection: Resolved
• Implement Connection based on AsyncConnection: Resolved
• Implement Admin based on AsyncAdmin: Resolved
• Rewrite RegionPlacementMaintainer to use AsyncClusterConnection: Resolved
• Remove ConnectionImplementation and related classes: Resolved
• Introduce a createClusterConnection method in ClusterConnectionFactory: Resolved
• Implement BufferedMutator Based on AsyncBufferedMutator: Resolved
• Remove the usage of the locateRegion related methods in ClusterConnection: Resolved
• Reimplement BulkLoadHFilesTool to use AsyncClusterConnection: Resolved
• Rewrite TestScannerHeartbeatMessages: Resolved
• Re-enable TestAvoidCellReferencesIntoShippedBlocks: Resolved
• Implement RegionLocator based on AsyncTableRegionLocator: Resolved
• Fix broken UTs on branch HBASE-21512: Resolved
• Fix TestRpcControllerFactory: Resolved
• Also catch RemoteException in SyncReplicationTestBase.verifyReplicationRequestRejection: Resolved
• Fix TestRegionServerCoprocessorEndpoint: Resolved
• Fix failed shell UTs: Resolved
• Fix TestClientOperationTimeout: Resolved
• Fix TestRegionMergeTransitionOnCluster and TestSplitTransactionOnCluster: Resolved
• Fix TestHbck: Resolved
• Fix TestReplicationDroppedTables: Resolved
• NPE in RegionReplicaReplicationEndpoint: Resolved
• Fix creating table timeout for TestProcedurePriority: Resolved
• RejectedExecutionException could be thrown from TableOverAsyncTable.coprocessor service if the connection has been shutown: Resolved
• Throw exception when creating thread pool if the connection has already been closed: Resolved
• NPE in RegionReplicaReplicationEndpoint: Resolved
• Compare performance number with old sync client implementation: Resolved
• BufferedMutatorOverAsyncBufferedMutator.tryCompleteFuture consume too much CPU time: Resolved