Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3510

Improve FSEditLog pre-allocation

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0, 2.0.0-alpha
    • 2.0.3-alpha
    • None
    • None
    • Reviewed

    Description

      It is good to avoid running out of space in the middle of writing a batch of edits, because when it happens, we often get partial edits at the end of the log.
      Edit log preallocation can solve this problem (see HADOOP-2330 for a full description of edit log preallocation).

      The current pre-allocation code was introduced for performance reasons, not for preventing partial edits. As a consequence, we sometimes do a write without using pre-allocation. We should change the pre-allocation code so that it always preallocates at least enough space before writing out the edits.

      Attachments

        1. HDFS-3510.001.patch
          6 kB
          Colin McCabe
        2. HDFS-3510.003.patch
          13 kB
          Colin McCabe
        3. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        4. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        5. HDFS-3510.006.patch
          14 kB
          Colin McCabe
        6. HDFS-3510.007.patch
          13 kB
          Colin McCabe
        7. HDFS-3510.008.patch
          13 kB
          Colin McCabe
        8. HDFS-3510.009.patch
          13 kB
          Colin McCabe
        9. HDFS-3510.010.patch
          22 kB
          Colin McCabe
        10. HDFS-3510.011.patch
          21 kB
          Colin McCabe
        11. HDFS-3510-b1.001.patch
          8 kB
          Colin McCabe
        12. HDFS-3510-b1.002.patch
          8 kB
          Colin McCabe
        13. hdfs-3510-branch-2.txt
          19 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmccabe Colin McCabe Assign to me
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment