Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2982

Startup performance suffers when there are many edit log segments

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.0-alpha
    • 2.0.2-alpha
    • namenode
    • None
    • Reviewed

    Description

      For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

      Attachments

        1. HDFS-2982.001.patch
          87 kB
          Colin McCabe
        2. HDFS-2982.002.patch
          84 kB
          Colin McCabe
        3. HDFS-2982.003.patch
          84 kB
          Colin McCabe
        4. HDFS-2982.004.patch
          84 kB
          Colin McCabe
        5. HDFS-2982.005.patch
          84 kB
          Colin McCabe
        6. HDFS-2982.006.patch
          87 kB
          Colin McCabe
        7. HDFS-2982.007.patch
          87 kB
          Colin McCabe
        8. HDFS-2982.008.patch
          87 kB
          Colin McCabe
        9. HDFS-2982.009.patch
          87 kB
          Colin McCabe

        Issue Links

          is superceded by

          New Feature - A new feature of the product, which has yet to be developed. HDFS-3049 During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              cmccabe Colin McCabe
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: