Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3335

check for edit log corruption at the end of the log

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Even after encountering an OP_INVALID, we should check the end of the edit log to make sure that it contains no more edits.

      This will catch things like rare race conditions or log corruptions that would otherwise remain undetected. They will got from being silent data loss scenarios to being cases that we can detect and fix.

      Using recovery mode, we can choose to ignore the end of the log if necessary.

      1. HDFS-3335.001.patch
        9 kB
        Colin Patrick McCabe
      2. HDFS-3335.002.patch
        23 kB
        Colin Patrick McCabe
      3. HDFS-3335.003.patch
        23 kB
        Colin Patrick McCabe
      4. HDFS-3335.004.patch
        24 kB
        Colin Patrick McCabe
      5. HDFS-3335.005.patch
        24 kB
        Colin Patrick McCabe
      6. HDFS-3335.006.patch
        25 kB
        Colin Patrick McCabe
      7. HDFS-3335.007.patch
        25 kB
        Colin Patrick McCabe
      8. HDFS-3335.008.patch
        23 kB
        Colin Patrick McCabe
      9. HDFS-3335.009.patch
        24 kB
        Colin Patrick McCabe
      10. HDFS-3335.010.patch
        24 kB
        Colin Patrick McCabe
      11. HDFS-3335.011.patch
        25 kB
        Colin Patrick McCabe
      12. HDFS-3335-b1.001.patch
        10 kB
        Colin Patrick McCabe
      13. HDFS-3335-b1.002.patch
        10 kB
        Colin Patrick McCabe
      14. HDFS-3335-b1.003.patch
        10 kB
        Colin Patrick McCabe
      15. HDFS-3335-b1.004.patch
        11 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Colin Patrick McCabe made changes -
          Link This issue relates to HDFS-3479 [ HDFS-3479 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Todd Lipcon made changes -
          Fix Version/s 2.0.1 [ 12321440 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.0 [ 12320353 ]
          Resolution Fixed [ 1 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.011.patch [ 12526723 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.010.patch [ 12526557 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.009.patch [ 12526556 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.008.patch [ 12526470 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335-b1.004.patch [ 12526442 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.007.patch [ 12526396 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.006.patch [ 12526074 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.005.patch [ 12525692 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.004.patch [ 12525691 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.003.patch [ 12525541 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.002.patch [ 12525355 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335.001.patch [ 12525247 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335-b1.003.patch [ 12525217 ]
          Colin Patrick McCabe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-3335-b1.002.patch [ 12525208 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Attachment HDFS-3335-b1.001.patch [ 12524954 ]
          Colin Patrick McCabe created issue -

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development