Hadoop Common
  1. Hadoop Common
  2. HADOOP-2703

New files under lease (before close) still shows up as MISSING files/blocks in fsck

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.15.3
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Changed fsck to ignore files opened for writing. Introduced new option "-openforwrite" to explicitly show open files.

      Description

      On 0.15.3 which includes HADOOP-2540 ("Empty blocks make fsck report corrupt, even when it isn't") fix, we are still seeing fsck reporting missing blocks when they aren't. Basically, any unclosed small files under lease shows up as MISSING.

      Can we have an option in fsck to skip those files under lease?
      Or count those files/blocks separately?

      1. HADOOP-2703-5.patch
        14 kB
        Lohit Vijayarenu
      2. HADOOP-2703-4.patch
        14 kB
        Lohit Vijayarenu
      3. HADOOP-2703-3.patch
        14 kB
        Lohit Vijayarenu
      4. HADOOP-2703-2.patch
        12 kB
        Lohit Vijayarenu
      5. HADOOP-2703-1.patch
        11 kB
        Lohit Vijayarenu

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Release Note This patch now ignores open files during fsck. A new option '-openforwrite' has been introduced to explicitly show open files. Changed fsck to ignore files opened for writing. Introduced new option "-openforwrite" to explicitly show open files.
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2703-5.patch [ 12380257 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2703-4.patch [ 12380220 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2703-3.patch [ 12380218 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note This patch now ignores open files during fsck. A new option '-open' has been introduced to explicitly show open files. This patch now ignores open files during fsck. A new option '-openforwrite' has been introduced to explicitly show open files.
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2703-2.patch [ 12379966 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Release Note This patch now ignores open files during fsck. A new option '-open' has been introduced to explicitly show open files.
          Hadoop Flags [Incompatible change]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2703-1.patch [ 12379934 ]
          Lohit Vijayarenu made changes -
          Assignee lohit vijayarenu [ lohit ]
          Lohit Vijayarenu made changes -
          Field Original Value New Value
          Link This issue is related to HADOOP-2540 [ HADOOP-2540 ]
          Koji Noguchi created issue -

            People

            • Assignee:
              Lohit Vijayarenu
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development