Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1117

DFS Scalability: When the namenode is restarted it consumes 80% CPU

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.12.0
    • 0.12.1
    • None
    • None

    Description

      When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

      Attachments

        1. CpuPendingTransfer3.patch
          2 kB
          Dhruba Borthakur

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-1113 namenode slowdown when orphan block(s) left in neededReplication

          • Major - Major loss of function.
          • Closed

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: