Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7217

Better batching of IBRs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After HDFS-2691 (pipeline recovery in HA), the number of IBR(incremental block report)s have doubled. Since processing of IBR requires exclusive FSNamesystem write lock, this can be a source of significant overhead on clusters with high write load.

      On one of the busy clusters, we have observed 60 to 70 percent of available handlers being constantly occupied by IBRs. This degrades throughput greatly when compared to 0.23.

        Attachments

        1. HDFS-7217.patch
          3 kB
          Kihwal Lee

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: