Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3510

Improve FSEditLog pre-allocation

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.0.0, 2.0.0-alpha
    • 2.0.3-alpha
    • None
    • None
    • Reviewed

    Description

      It is good to avoid running out of space in the middle of writing a batch of edits, because when it happens, we often get partial edits at the end of the log.
      Edit log preallocation can solve this problem (see HADOOP-2330 for a full description of edit log preallocation).

      The current pre-allocation code was introduced for performance reasons, not for preventing partial edits. As a consequence, we sometimes do a write without using pre-allocation. We should change the pre-allocation code so that it always preallocates at least enough space before writing out the edits.

      Attachments

        1. HDFS-3510.001.patch
          6 kB
          Colin McCabe
        2. HDFS-3510.003.patch
          13 kB
          Colin McCabe
        3. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        4. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        5. HDFS-3510.006.patch
          14 kB
          Colin McCabe
        6. HDFS-3510.007.patch
          13 kB
          Colin McCabe
        7. HDFS-3510.008.patch
          13 kB
          Colin McCabe
        8. HDFS-3510.009.patch
          13 kB
          Colin McCabe
        9. HDFS-3510.010.patch
          22 kB
          Colin McCabe
        10. HDFS-3510.011.patch
          21 kB
          Colin McCabe
        11. HDFS-3510-b1.001.patch
          8 kB
          Colin McCabe
        12. HDFS-3510-b1.002.patch
          8 kB
          Colin McCabe
        13. hdfs-3510-branch-2.txt
          19 kB
          Todd Lipcon

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmccabe Colin McCabe
            cmccabe Colin McCabe
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment