Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14403

Cost-Based RPC FairCallQueue

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations).

      This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.
      Show
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations). This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.

      Description

      HADOOP-15016 initially described extensions to the Hadoop FairCallQueue encompassing both cost-based analysis of incoming RPCs, as well as support for reservations of RPC capacity for system/platform users. This JIRA intends to track the former, as HADOOP-15016 was repurposed to more specifically focus on the reservation portion of the work.

        Attachments

        1. CostBasedFairCallQueueDesign_v0.pdf
          1.09 MB
          Erik Krogen
        2. HDFS-14403.001.patch
          62 kB
          Christopher Gregorian
        3. HDFS-14403.002.patch
          63 kB
          Christopher Gregorian
        4. HDFS-14403.003.patch
          63 kB
          Christopher Gregorian
        5. HDFS-14403.004.patch
          33 kB
          Christopher Gregorian
        6. HDFS-14403.005.patch
          44 kB
          Christopher Gregorian
        7. HDFS-14403.006.combined.patch
          54 kB
          Christopher Gregorian
        8. HDFS-14403.006.patch
          17 kB
          Christopher Gregorian
        9. HDFS-14403.007.patch
          15 kB
          Erik Krogen
        10. HDFS-14403.008.patch
          34 kB
          Erik Krogen
        11. HDFS-14403.009.patch
          48 kB
          Erik Krogen
        12. HDFS-14403.010.patch
          48 kB
          Erik Krogen
        13. HDFS-14403.011.patch
          47 kB
          Erik Krogen
        14. HDFS-14403.012.patch
          55 kB
          Erik Krogen
        15. HDFS-14403.013.patch
          55 kB
          Erik Krogen
        16. HDFS-14403.branch-2.8.patch
          34 kB
          Daryn Sharp

          Issue Links

            Activity

              People

              • Assignee:
                cgregori Christopher Gregorian
                Reporter:
                xkrogen Erik Krogen
              • Votes:
                0 Vote for this issue
                Watchers:
                28 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: