Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2718

Optimize OP_ADD in edits loading

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.0, 1.0.0, 2.0.0-alpha
    • 0.22.1
    • namenode
    • None

    Description

      During loading the edits journal FSEditLog.loadEditRecords() processes OP_ADD inefficiently. It first removes the existing INodeFile from the directory tree, then adds it back as a regular INodeFile, and then replaces it with INodeFileUnderConstruction if files is not closed. This slows down edits loading. OP_ADD should be done in one shot and retain previously existing data.

      Attachments

        1. editsLoader-trunk.patch
          16 kB
          Konstantin Shvachko
        2. editsLoader-trunk.patch
          17 kB
          Konstantin Shvachko
        3. editsLoader-trunk.patch
          18 kB
          Konstantin Shvachko
        4. editsLoader-trunk.patch
          18 kB
          Konstantin Shvachko
        5. editsLoader-0.22.patch
          18 kB
          Konstantin Shvachko
        6. editsLoader-0.22.patch
          20 kB
          Konstantin Shvachko
        7. editsLoader-0.22.patch
          20 kB
          Konstantin Shvachko

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shv Konstantin Shvachko
            shv Konstantin Shvachko
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment