Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11552

Allow handoff on the server side for RPC requests

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha2
    • ipc
    • None
    • Reviewed

    Description

      An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism.

      Some use cases where this is useful.

      • YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call.
      • YARN allocate - requests and allocations use the same protocol. New allocations are received via polling.
        The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available.
        MapReduce/Tez task to AM communication is another example of this pattern.

      The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently.

      I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN.

      Attachments

        1. HADOOP-11552.05.patch
          38 kB
          Siddharth Seth
        2. HADOOP-11552.06.patch
          39 kB
          Siddharth Seth
        3. HADOOP-11552.07.patch
          39 kB
          Siddharth Seth
        4. HADOOP-11552.08.patch
          39 kB
          Jian He
        5. HADOOP-11552.1.wip.txt
          18 kB
          Siddharth Seth
        6. HADOOP-11552.2.txt
          30 kB
          Siddharth Seth
        7. HADOOP-11552.3.txt
          31 kB
          Siddharth Seth
        8. HADOOP-11552.3.txt
          30 kB
          Siddharth Seth
        9. HADOOP-11552.4.txt
          30 kB
          Siddharth Seth

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            44 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment