Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2982

Startup performance suffers when there are many edit log segments

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.0.2-alpha
    • namenode
    • None
    • Reviewed

    Description

      For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

      Attachments

        1. HDFS-2982.009.patch
          87 kB
          Colin McCabe
        2. HDFS-2982.008.patch
          87 kB
          Colin McCabe
        3. HDFS-2982.007.patch
          87 kB
          Colin McCabe
        4. HDFS-2982.006.patch
          87 kB
          Colin McCabe
        5. HDFS-2982.005.patch
          84 kB
          Colin McCabe
        6. HDFS-2982.004.patch
          84 kB
          Colin McCabe
        7. HDFS-2982.003.patch
          84 kB
          Colin McCabe
        8. HDFS-2982.002.patch
          84 kB
          Colin McCabe
        9. HDFS-2982.001.patch
          87 kB
          Colin McCabe

        Issue Links

          Activity

            People

              cmccabe Colin McCabe
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: