Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17531

RBF: Asynchronous router RPC

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • rbf

    Description

      Description

      Currently, the main function of the Router service is to accept client requests, forward the requests to the corresponding downstream ns, and then return the results of the downstream ns to the client. The link is as follows:


      The main threads involved in the rpc link are:
      Read: Get the client request and put it into the call queue (1)
      Handler:
      Extract call (2) from the call queue, process the call, generate a new call, place it in the call of the connection thread, and wait for the call processing to complete (3)
      After being awakened by the connection thread, process the response and put it into the response queue (5)
      Connection:
      Hold the link with downstream ns, send the call from the call to the downstream ns (via rpcRequestThread), and obtain a response from ns. Based on the call in the response, notify the call to complete processing (4)
      Responder:
      Retrieve the response queue from the queue (6) and return it to the client
       

      Shortcoming
      Even if the connection thread can send more requests to downstream nameservices, since (3) and (4) are synchronous, when the handler thread adds the call to connection.calls, it needs to wait until the connection notifies the call to complete, and then Only after the response is put into the response queue can a new call be obtained from the call queue and processed. Therefore, the concurrency performance of the router is limited by the number of handlers; a simple example is as follows: If the number of handlers is 1 and the maximum number of calls in the connection thread is 10, then even if the connection thread can send 10 requests to the downstream ns, since the number of handlers is 1, the router can only process one request after another. 
       
      Since the performance of router rpc is mainly limited by the number of handlers, the most effective way to improve rpc performance currently is to increase the number of handlers. Letting the router create a large number of handler threads will also increase the number of thread switches and cannot maximize the use of machine performance.
       
      There are usually multiple ns downstream of the router. If the handler forwards the request to an ns with poor performance, it will cause the handler to wait for a long time. Due to the reduction of available handlers, the router's ability to handle ns requests with normal performance will be reduced. From the perspective of the client, the performance of the downstream ns of the router has deteriorated at this time. We often find that the call queue of the downstream ns is not high, but the call queue of the router is very high.
       
      Therefore, although the main function of the router is to federate and handle requests from multiple NSs, the current synchronous RPC performance cannot satisfy the scenario where there are many NSs downstream of the router. Even if the concurrent performance of the router can be improved by increasing the number of handlers, it is still relatively slow. More threads will increase the CPU context switching time, and in fact many of the handler threads are in a blocked state, which is undoubtedly a waste of thread resources. When a request enters the router, there is no guarantee that there will be a running handler at this time.
       

      Therefore, I consider asynchronous router rpc. Please view the pdf for the complete solution.

       

      Welcome everyone to exchange and discuss!

      Attachments

        1. Router asynchronous rpc implementation .pdf
          611 kB
          Jian Zhang
        2. image-2024-05-19-18-07-51-282.png
          185 kB
          Jian Zhang
        3. HDFS-17531.001.patch
          505 kB
          Jian Zhang
        4. Comparison of Async router & sync router performance.pdf
          10.68 MB
          Jian Zhang
        5. Aynchronous router.pdf
          165 kB
          Jian Zhang
        6. Async router single ns performance test.pdf
          2.77 MB
          Jian Zhang

        Issue Links

          Activity

            People

              keepromise Jian Zhang
              keepromise Jian Zhang
              Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated: