Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3510

Improve FSEditLog pre-allocation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It is good to avoid running out of space in the middle of writing a batch of edits, because when it happens, we often get partial edits at the end of the log.
      Edit log preallocation can solve this problem (see HADOOP-2330 for a full description of edit log preallocation).

      The current pre-allocation code was introduced for performance reasons, not for preventing partial edits. As a consequence, we sometimes do a write without using pre-allocation. We should change the pre-allocation code so that it always preallocates at least enough space before writing out the edits.

        Attachments

        1. HDFS-3510.001.patch
          6 kB
          Colin McCabe
        2. HDFS-3510.003.patch
          13 kB
          Colin McCabe
        3. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        4. HDFS-3510.004.patch
          13 kB
          Colin McCabe
        5. HDFS-3510.006.patch
          14 kB
          Colin McCabe
        6. HDFS-3510.007.patch
          13 kB
          Colin McCabe
        7. HDFS-3510.008.patch
          13 kB
          Colin McCabe
        8. HDFS-3510.009.patch
          13 kB
          Colin McCabe
        9. HDFS-3510.010.patch
          22 kB
          Colin McCabe
        10. HDFS-3510.011.patch
          21 kB
          Colin McCabe
        11. HDFS-3510-b1.001.patch
          8 kB
          Colin McCabe
        12. HDFS-3510-b1.002.patch
          8 kB
          Colin McCabe
        13. hdfs-3510-branch-2.txt
          19 kB
          Todd Lipcon

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin McCabe
                Reporter:
                cmccabe Colin McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: