Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21512

Reimplement sync client based on async client

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha-1
    • Component/s: asyncclient, Client
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Reimplement sync client based on async client, the old code for sync client have been purged.

      Most public APIs are kept, except:

      1. coprocessor related methods in sync client interface have been marked as deprecated. This is because that these methods may reference methods from the BlockingInterface of a protobuf stub, so we have to use a thread pool to execute these methods which are not good. Please use the coprocessor methods in async client interface instead.

      2. BufferedMutatorParams.setWriteBufferPeriodicFlushTimerTickMs has been marked as deprecated. It has no effect now, just kept for compatibility. We use a global timer per connection so you can not customize the tick any more.

      3. BufferedMutatorParams.implementationClassName has been marked as deprecated. It has no effect now, as we will pass several internal stuff to the constructor when creating the implementation instance, which is not good. We should find another way to do it.

      4. The behavior for the two Admin.split methods is changed a bit. Now when the methods return, the split should have been done, while in the past it just means we have send the request out.

      5. Now most(non-deprecated) methods in Admin and Table interface do not accept null values any more.

      6. LoadIncrementalHFiles has been removed. It was marked as deprecated in 2.2.0 so by default is should only be removed in 4.0.0. But it is an implementation class, neither an interface nor a data structure, it is not a good practice to make it public as it exposes several internal states, and is also easily to be misused in the future, so we just remove it in 3.0.0. Users can use BulkLoadHFiles in your code, and there is no difference when executing bulk load from command line.

      And we also introduce a toAsyncConnection method in the Connection interface, and also a toConnection method in the AsyncConnection interface. They are both lightweighted operations, the returned instance will share the same connection to the hbase cluster, so you can easily convert Connection to AsyncConnection and vice visa.

      For internal developers, the ClusterConnection has also been purged. Please use the new AsyncClusterConnection instead, and please also see the AsyncRegionServerAdmin, which wraps the call to a region server. There is no sync interface any more, please use FutureUtils.get on the returned Future if you want the sync behavior. And also the ShortCircultConnection has been removed, as now we will use async stub but our RpcServices only implement BlockingInterface.
      Show
      Reimplement sync client based on async client, the old code for sync client have been purged. Most public APIs are kept, except: 1. coprocessor related methods in sync client interface have been marked as deprecated. This is because that these methods may reference methods from the BlockingInterface of a protobuf stub, so we have to use a thread pool to execute these methods which are not good. Please use the coprocessor methods in async client interface instead. 2. BufferedMutatorParams.setWriteBufferPeriodicFlushTimerTickMs has been marked as deprecated. It has no effect now, just kept for compatibility. We use a global timer per connection so you can not customize the tick any more. 3. BufferedMutatorParams.implementationClassName has been marked as deprecated. It has no effect now, as we will pass several internal stuff to the constructor when creating the implementation instance, which is not good. We should find another way to do it. 4. The behavior for the two Admin.split methods is changed a bit. Now when the methods return, the split should have been done, while in the past it just means we have send the request out. 5. Now most(non-deprecated) methods in Admin and Table interface do not accept null values any more. 6. LoadIncrementalHFiles has been removed. It was marked as deprecated in 2.2.0 so by default is should only be removed in 4.0.0. But it is an implementation class, neither an interface nor a data structure, it is not a good practice to make it public as it exposes several internal states, and is also easily to be misused in the future, so we just remove it in 3.0.0. Users can use BulkLoadHFiles in your code, and there is no difference when executing bulk load from command line. And we also introduce a toAsyncConnection method in the Connection interface, and also a toConnection method in the AsyncConnection interface. They are both lightweighted operations, the returned instance will share the same connection to the hbase cluster, so you can easily convert Connection to AsyncConnection and vice visa. For internal developers, the ClusterConnection has also been purged. Please use the new AsyncClusterConnection instead, and please also see the AsyncRegionServerAdmin, which wraps the call to a region server. There is no sync interface any more, please use FutureUtils.get on the returned Future if you want the sync behavior. And also the ShortCircultConnection has been removed, as now we will use async stub but our RpcServices only implement BlockingInterface.

      Description

      At least for the RSProcedureDispatcher, with CompletableFuture we do not need to set a delay and use a thread pool any more, which could reduce the resource usage and also the latency.

      Once this is done, I think we can remove the ClusterConnection completely, and start to rewrite the old sync client based on the async client, which could reduce the code base a lot for our client.

        Attachments

          Issue Links

          1.
          Also initialize an AsyncClusterConnection in HRegionServer Sub-task Resolved Duo Zhang
          2.
          Use AsyncConnection instead of Connection in SecureBulkLoadManager Sub-task Resolved Duo Zhang
          3.
          Use AsyncClusterConnection in ServerManager for getRsAdmin Sub-task Resolved Duo Zhang
          4.
          Rewrite ServerManager.closeRegionSilentlyAndWait to use AsyncClusterConnection Sub-task Resolved Duo Zhang
          5.
          Rewrite RegionReplicaFlushHandler to use AsyncClusterConnection Sub-task Resolved Duo Zhang
          6.
          Use AsyncClusterConnection for HBaseInterClusterReplicationEndpoint Sub-task Resolved Duo Zhang
          7.
          Remove ClusterConnection Sub-task Resolved Duo Zhang
          8.
          Rewrite RegionReplicaReplicationEndpoint to use AsyncClusterConnection Sub-task Resolved Duo Zhang
          9.
          Implement Connection based on AsyncConnection Sub-task Resolved Duo Zhang
          10.
          Implement Admin based on AsyncAdmin Sub-task Resolved Duo Zhang
          11.
          Rewrite RegionPlacementMaintainer to use AsyncClusterConnection Sub-task Resolved Duo Zhang
          12.
          Remove ConnectionImplementation and related classes Sub-task Resolved Duo Zhang
          13.
          Introduce a createClusterConnection method in ClusterConnectionFactory Sub-task Resolved Unassigned
          14.
          Implement BufferedMutator Based on AsyncBufferedMutator Sub-task Resolved Duo Zhang
          15.
          Remove the usage of the locateRegion related methods in ClusterConnection Sub-task Resolved Duo Zhang
          16.
          Reimplement BulkLoadHFilesTool to use AsyncClusterConnection Sub-task Resolved Duo Zhang
          17.
          Rewrite TestScannerHeartbeatMessages Sub-task Resolved Duo Zhang
          18.
          Re-enable TestAvoidCellReferencesIntoShippedBlocks Sub-task Resolved Duo Zhang
          19.
          Implement RegionLocator based on AsyncTableRegionLocator Sub-task Resolved Duo Zhang
          20.
          Fix broken UTs on branch HBASE-21512 Sub-task Resolved Duo Zhang
          21.
          Fix TestRpcControllerFactory Sub-task Resolved Duo Zhang
          22.
          Also catch RemoteException in SyncReplicationTestBase.verifyReplicationRequestRejection Sub-task Resolved Duo Zhang
          23.
          Fix TestRegionServerCoprocessorEndpoint Sub-task Resolved Duo Zhang
          24.
          Fix failed shell UTs Sub-task Resolved Duo Zhang
          25.
          Fix TestClientOperationTimeout Sub-task Resolved Duo Zhang
          26.
          Fix TestRegionMergeTransitionOnCluster and TestSplitTransactionOnCluster Sub-task Resolved Duo Zhang
          27.
          Fix TestHbck Sub-task Resolved Duo Zhang
          28.
          Fix TestReplicationDroppedTables Sub-task Resolved Duo Zhang
          29.
          NPE in RegionReplicaReplicationEndpoint Sub-task Resolved Duo Zhang
          30.
          Fix creating table timeout for TestProcedurePriority Sub-task Resolved Duo Zhang
          31.
          RejectedExecutionException could be thrown from TableOverAsyncTable.coprocessor service if the connection has been shutown Sub-task Resolved Duo Zhang
          32.
          Throw exception when creating thread pool if the connection has already been closed Sub-task Resolved Duo Zhang
          33.
          NPE in RegionReplicaReplicationEndpoint Sub-task Resolved Duo Zhang
          34.
          Compare performance number with old sync client implementation Sub-task Resolved Duo Zhang
          35.
          BufferedMutatorOverAsyncBufferedMutator.tryCompleteFuture consume too much CPU time Sub-task Resolved Duo Zhang

            Activity

              People

              • Assignee:
                zhangduo Duo Zhang
                Reporter:
                zhangduo Duo Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: