Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2982

Startup performance suffers when there are many edit log segments

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

      1. HDFS-2982.009.patch
        87 kB
        Colin Patrick McCabe
      2. HDFS-2982.008.patch
        87 kB
        Colin Patrick McCabe
      3. HDFS-2982.007.patch
        87 kB
        Colin Patrick McCabe
      4. HDFS-2982.006.patch
        87 kB
        Colin Patrick McCabe
      5. HDFS-2982.005.patch
        84 kB
        Colin Patrick McCabe
      6. HDFS-2982.004.patch
        84 kB
        Colin Patrick McCabe
      7. HDFS-2982.003.patch
        84 kB
        Colin Patrick McCabe
      8. HDFS-2982.002.patch
        84 kB
        Colin Patrick McCabe
      9. HDFS-2982.001.patch
        87 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development