Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1371

One bad node can incorrectly flag many files as corrupt

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Environment:

      yahoo internal version
      [knoguchi@gwgd4003 ~]$ hadoop version
      Hadoop 0.20.104.3.1007030707

    • Hadoop Flags:
      Reviewed

      Description

      On our cluster, 12 files were reported as corrupt by fsck even though the replicas on the datanodes were healthy.
      Turns out that all the replicas (12 files x 3 replicas per file) were reported corrupt from one node.

      Surprisingly, these files were still readable/accessible from dfsclient (-get/-cat) without any problems.

      1. HDFS-1371.0518.2.patch
        27 kB
        Tanping Wang
      2. HDFS-1371.0518.patch
        13 kB
        Tanping Wang
      3. HDFS-1371.0517.2.patch
        27 kB
        Tanping Wang
      4. HDFS-1371.0517.patch
        27 kB
        Tanping Wang
      5. HDFS-1371.0515.patch
        27 kB
        Tanping Wang
      6. HDFS-1371.0513.patch
        27 kB
        Tanping Wang
      7. HDFS-1371.0503.patch
        21 kB
        Tanping Wang
      8. HDFS-1371.04252011.patch
        25 kB
        Tanping Wang

        Activity

        Jitendra Nath Pandey made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Tanping Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tanping Wang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tanping Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0518.2.patch [ 12479689 ]
        Tanping Wang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tanping Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0518.patch [ 12479660 ]
        Tanping Wang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Jitendra Nath Pandey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags [Reviewed]
        Affects Version/s 0.23.0 [ 12315571 ]
        Fix Version/s 0.23.0 [ 12315571 ]
        Jitendra Nath Pandey made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0517.2.patch [ 12479533 ]
        Tanping Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0517.patch [ 12479526 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0517.patch [ 12479525 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0517.patch [ 12479525 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0515.patch [ 12479352 ]
        Tanping Wang made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tanping Wang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0513.patch [ 12479061 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0503.patch [ 12478008 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0502.patch [ 12477989 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0502.patch [ 12477989 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0502.patch [ 12477986 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.0502.patch [ 12477986 ]
        Tanping Wang made changes -
        Attachment HDFS-1371.04252011.patch [ 12477335 ]
        Tanping Wang made changes -
        Field Original Value New Value
        Assignee Tanping Wang [ tanping ]
        Koji Noguchi created issue -

          People

          • Assignee:
            Tanping Wang
            Reporter:
            Koji Noguchi
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development