Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2126 Improve Namenode startup time [umbrella task]
  3. HDFS-1391

Exiting safemode takes a long time when there are lots of blocks in the HDFS

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      When the namenode decides to exit safemode, it acquires the FSNamesystem lock and then iterates over all blocks in the blocksmap to determine if any block has any excess replicas. This call takes upwards of 5 minutes on a cluster that has 100 million blocks. This delays namenode restart to a good extent.

        Attachments

        1. excessReplicas.1_trunk.txt
          16 kB
          dhruba borthakur
        2. excessReplicas2.txt
          16 kB
          dhruba borthakur
        3. excessReplicas3.txt
          18 kB
          dhruba borthakur
        4. excessReplicas5.txt
          19 kB
          dhruba borthakur
        5. 1391_excessReplicas5_comments.txt
          4 kB
          Matt Foley

          Activity

            People

            • Assignee:
              dhruba dhruba borthakur
              Reporter:
              dhruba dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated: