Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9194

RPC Support for QoS

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.2-alpha
    • 2.1.0-beta
    • ipc
    • None
    • Incompatible change, Reviewed
    • Part of the RPC version 9 change. A service class byte is added after the version byte.

    Description

      One of the next frontiers of Hadoop performance is QoS (Quality of Service). We need QoS support to fight the inevitable "buffer bloat" (including various queues, which are probably necessary for throughput) in our software stack. This is important for mixed workload with different latency and throughput requirements (e.g. OLTP vs OLAP, batch and even compaction I/O) against the same DFS.

      Any potential bottleneck will need to be managed by QoS mechanisms, starting with RPC.

      How about adding a one byte DS (differentiated services) field (a la the 6-bit DS field in IP header) in the RPC header to facilitate the QoS mechanisms (in separate JIRAs)? The byte at a fixed offset (how about 0?) of the header is helpful for implementing high performance QoS mechanisms in switches (software or hardware) and servers with minimum decoding effort.

      Attachments

        1. HADOOP-9194.patch
          8 kB
          Junping Du
        2. HADOOP-9194-v2.patch
          13 kB
          Junping Du

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            junping_du Junping Du
            vicaya Luke Lu
            Votes:
            0 Vote for this issue
            Watchers:
            36 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment