Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14403

Cost-Based RPC FairCallQueue

    XMLWordPrintableJSON

Details

    • Hide
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations).

      This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.
      Show
      This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user's operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations). This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.

    Description

      HADOOP-15016 initially described extensions to the Hadoop FairCallQueue encompassing both cost-based analysis of incoming RPCs, as well as support for reservations of RPC capacity for system/platform users. This JIRA intends to track the former, as HADOOP-15016 was repurposed to more specifically focus on the reservation portion of the work.

      Attachments

        1. CostBasedFairCallQueueDesign_v0.pdf
          1.09 MB
          Erik Krogen
        2. HDFS-14403.001.patch
          62 kB
          Christopher Gregorian
        3. HDFS-14403.002.patch
          63 kB
          Christopher Gregorian
        4. HDFS-14403.003.patch
          63 kB
          Christopher Gregorian
        5. HDFS-14403.004.patch
          33 kB
          Christopher Gregorian
        6. HDFS-14403.005.patch
          44 kB
          Christopher Gregorian
        7. HDFS-14403.006.combined.patch
          54 kB
          Christopher Gregorian
        8. HDFS-14403.006.patch
          17 kB
          Christopher Gregorian
        9. HDFS-14403.007.patch
          15 kB
          Erik Krogen
        10. HDFS-14403.008.patch
          34 kB
          Erik Krogen
        11. HDFS-14403.009.patch
          48 kB
          Erik Krogen
        12. HDFS-14403.010.patch
          48 kB
          Erik Krogen
        13. HDFS-14403.011.patch
          47 kB
          Erik Krogen
        14. HDFS-14403.012.patch
          55 kB
          Erik Krogen
        15. HDFS-14403.013.patch
          55 kB
          Erik Krogen
        16. HDFS-14403.branch-2.8.patch
          34 kB
          Daryn Sharp

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16266 Add more fine-grained processing time metrics to the RPC layer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-15016 RPC FairCallQueue Reservation Support

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-14667 Backport [HDFS-14403] "Cost-based FairCallQueue" to branch-2

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              cgregori Christopher Gregorian
              xkrogen Erik Krogen
              Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: