Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2982

Startup performance suffers when there are many edit log segments

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

        Attachments

        1. HDFS-2982.001.patch
          87 kB
          Colin P. McCabe
        2. HDFS-2982.002.patch
          84 kB
          Colin P. McCabe
        3. HDFS-2982.003.patch
          84 kB
          Colin P. McCabe
        4. HDFS-2982.004.patch
          84 kB
          Colin P. McCabe
        5. HDFS-2982.005.patch
          84 kB
          Colin P. McCabe
        6. HDFS-2982.006.patch
          87 kB
          Colin P. McCabe
        7. HDFS-2982.007.patch
          87 kB
          Colin P. McCabe
        8. HDFS-2982.008.patch
          87 kB
          Colin P. McCabe
        9. HDFS-2982.009.patch
          87 kB
          Colin P. McCabe

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: