Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5346

Avoid unnecessary call to getNumLiveDataNodes() for each block during IBR processing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.9, 2.3.0
    • Fix Version/s: 0.23.10, 2.3.0
    • Component/s: namenode, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When initial block reports are being processed, checkMode() is called from incrementSafeBlockCount(). This causes the replication queues to be initialized in the middle of processing a block report in the IBR processing mode. If there are many block reports waiting to be processed, SafeModeMonitor won't be able to make name node leave the safe mode soon. It appears that the block report processing speed degrades considerably during this time.

      Update: The main issue can be resolved by config. The other issue of calling getNumLiveDataNodes() for each block in the block report will be addressed in this jira

        Attachments

        1. HDFS-5346.patch
          0.8 kB
          Ravi Prakash
        2. HDFS-5346.branch-23.patch
          0.8 kB
          Ravi Prakash
        3. HDFS-5346.patch
          3 kB
          Ravi Prakash
        4. HDFS-5346.branch-23.patch
          3 kB
          Ravi Prakash
        5. HDFS-5346.patch
          2 kB
          Ravi Prakash

          Activity

            People

            • Assignee:
              raviprak Ravi Prakash
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: