Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1111

getCorruptFiles() should give some hint that the list is not complete

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If the list of corruptfiles returned by the namenode doesn't say anything if the number of corrupted files is larger than the call output limit (which means the list is not complete). There should be a way to hint incompleteness to clients.

      A simple hack would be to add an extra entry to the array returned with the value null. Clients could interpret this as a sign that there are other corrupt files in the system.

      We should also do some rephrasing of the fsck output to make it more confident when the list is not complete and less confident when the list is known to be incomplete.

      1. HADFS-1111.0.patch
        23 kB
        Rodrigo Schmidt
      2. HDFS-1111.trunk.1.patch
        55 kB
        Sriram Rao
      3. HDFS-1111.trunk.2.patch
        55 kB
        Sriram Rao
      4. HDFS-1111.trunk.2s.patch
        56 kB
        Konstantin Shvachko
      5. HDFS-1111.trunk.patch
        39 kB
        Sriram Rao
      6. HDFS-1111-y20.1.patch
        16 kB
        Sriram Rao
      7. HDFS-1111-y20.2.patch
        23 kB
        Sriram Rao

        Issue Links

          Activity

          Rodrigo Schmidt created issue -
          Rodrigo Schmidt made changes -
          Field Original Value New Value
          Assignee Rodrigo Schmidt [ rschmidt ]
          Rodrigo Schmidt made changes -
          Link This issue depends on HDFS-729 [ HDFS-729 ]
          Rodrigo Schmidt made changes -
          Link This issue depends on HDFS-1032 [ HDFS-1032 ]
          Rodrigo Schmidt made changes -
          Link This issue is related to HDFS-1031 [ HDFS-1031 ]
          Rodrigo Schmidt made changes -
          Link This issue is blocked by HADOOP-6743 [ HADOOP-6743 ]
          Rodrigo Schmidt made changes -
          Attachment HADFS-1111.0.patch [ 12446698 ]
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rodrigo Schmidt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rodrigo Schmidt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rodrigo Schmidt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Rodrigo Schmidt made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Rodrigo Schmidt made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Shvachko made changes -
          Link This issue is related to HDFS-1265 [ HDFS-1265 ]
          Sriram Rao made changes -
          Attachment HDFS-1111-y20.1.patch [ 12450965 ]
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Sriram Rao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sriram Rao made changes -
          Attachment HDFS-1111-y20.2.patch [ 12451460 ]
          Sriram Rao made changes -
          Attachment HDFS-1111-y20.2.patch [ 12451460 ]
          Sriram Rao made changes -
          Attachment HDFS-1111-y20.2.patch [ 12451461 ]
          Konstantin Shvachko made changes -
          Assignee Rodrigo Schmidt [ rschmidt ] Sriram Rao [ sriramsrao ]
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Sriram Rao made changes -
          Affects Version/s 0.22.0 [ 12314241 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.patch [ 12452993 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.patch [ 12452993 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.patch [ 12452997 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.1.patch [ 12453481 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.patch [ 12452997 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.patch [ 12453482 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.1.patch [ 12453481 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.1.patch [ 12453483 ]
          Sriram Rao made changes -
          Attachment HDFS-1111.trunk.2.patch [ 12454262 ]
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Shvachko made changes -
          Attachment HDFS-1111.trunk.2s.patch [ 12454386 ]
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends on HDFS-729 [ HDFS-729 ]
          Gavin made changes -
          Link This issue depends upon HDFS-729 [ HDFS-729 ]
          Gavin made changes -
          Link This issue depends on HDFS-1032 [ HDFS-1032 ]
          Gavin made changes -
          Link This issue depends upon HDFS-1032 [ HDFS-1032 ]

            People

            • Assignee:
              Sriram Rao
              Reporter:
              Rodrigo Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development