Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12140

Remove BPOfferService lock contention to get block pool id

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The block pool id is protected by a lock in BPOfferService. This creates excessive contention especially for xceivers threads attempting to queue IBRs and heartbeat processing. When the latter is delayed due to excessive FSDataset lock contention, it causes pipelines to collapse.

      Accessing the block pool id should be lockless after registration.

        Attachments

        1. HDFS-12140.trunk.patch
          5 kB
          Daryn Sharp
        2. HDFS-12140.branch-2.8.patch
          5 kB
          Daryn Sharp

          Activity

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: