Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2982

Startup performance suffers when there are many edit log segments

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

      1. HDFS-2982.001.patch
        87 kB
        Colin Patrick McCabe
      2. HDFS-2982.002.patch
        84 kB
        Colin Patrick McCabe
      3. HDFS-2982.003.patch
        84 kB
        Colin Patrick McCabe
      4. HDFS-2982.004.patch
        84 kB
        Colin Patrick McCabe
      5. HDFS-2982.005.patch
        84 kB
        Colin Patrick McCabe
      6. HDFS-2982.006.patch
        87 kB
        Colin Patrick McCabe
      7. HDFS-2982.007.patch
        87 kB
        Colin Patrick McCabe
      8. HDFS-2982.008.patch
        87 kB
        Colin Patrick McCabe
      9. HDFS-2982.009.patch
        87 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Todd Lipcon created issue -
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Link This issue is superceded by HDFS-3049 [ HDFS-3049 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.001.patch [ 12527992 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eli Collins made changes -
          Affects Version/s 2.0.0 [ 12320353 ]
          Affects Version/s 0.24.0 [ 12317653 ]
          Target Version/s HA branch (HDFS-1623), 0.24.0 [ 12317568, 12317653 ] 2.0.1 [ 12321440 ]
          Eli Collins made changes -
          Assignee Todd Lipcon [ tlipcon ] Colin Patrick McCabe [ cmccabe ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.002.patch [ 12528009 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.003.patch [ 12528117 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.004.patch [ 12528123 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.005.patch [ 12528152 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.006.patch [ 12528463 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.007.patch [ 12528468 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.008.patch [ 12528506 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-2982.009.patch [ 12528535 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development