Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14403

Cost-Based RPC FairCallQueue

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations).

      This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.
      Show
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations). This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.

    Description

      HADOOP-15016 initially described extensions to the Hadoop FairCallQueue encompassing both cost-based analysis of incoming RPCs, as well as support for reservations of RPC capacity for system/platform users. This JIRA intends to track the former, as HADOOP-15016 was repurposed to more specifically focus on the reservation portion of the work.

      Attachments

        1. HDFS-14403.013.patch
          55 kB
          Erik Krogen
        2. HDFS-14403.012.patch
          55 kB
          Erik Krogen
        3. HDFS-14403.011.patch
          47 kB
          Erik Krogen
        4. HDFS-14403.010.patch
          48 kB
          Erik Krogen
        5. HDFS-14403.009.patch
          48 kB
          Erik Krogen
        6. HDFS-14403.008.patch
          34 kB
          Erik Krogen
        7. HDFS-14403.007.patch
          15 kB
          Erik Krogen
        8. HDFS-14403.006.combined.patch
          54 kB
          Christopher Gregorian
        9. HDFS-14403.006.patch
          17 kB
          Christopher Gregorian
        10. HDFS-14403.005.patch
          44 kB
          Christopher Gregorian
        11. HDFS-14403.004.patch
          33 kB
          Christopher Gregorian
        12. HDFS-14403.branch-2.8.patch
          34 kB
          Daryn Sharp
        13. HDFS-14403.003.patch
          63 kB
          Christopher Gregorian
        14. HDFS-14403.002.patch
          63 kB
          Christopher Gregorian
        15. HDFS-14403.001.patch
          62 kB
          Christopher Gregorian
        16. CostBasedFairCallQueueDesign_v0.pdf
          1.09 MB
          Erik Krogen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cgregori Christopher Gregorian
            xkrogen Erik Krogen
            Votes:
            0 Vote for this issue
            Watchers:
            32 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment