Hadoop Common
  1. Hadoop Common
  2. HADOOP-4616

assertion makes fuse-dfs exit when reading incomplete data

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.2
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When trying to read a file that is corrupt on HDFS (registered by the namenode, but part of the data is missing on the datanodes), some of the assertions in dfs_read fail, causing the program to abort. This makes it impossible to access the mounted partition until it is mounted again.

      A simple way to reproduce this bug is to remove enough datanodes to have part of the data missing, and to read each file listed in HDFS.

      this is the assertion that fails (fuse_dfs.c:903) : assert(bufferReadIndex >= 0 && bufferReadIndex < fh->bufferSize);

      The expected behaviour would be to return either no file or a corrupt file, but continue working afterward.

      removing the assertion seems to work for now, but a special behaviour is probably needed to handle this particular problem correctly.

      1. HADOOP-4616.txt
        2 kB
        Pete Wyckoff
      2. HADOOP-4616.txt
        2 kB
        Pete Wyckoff
      3. HADOOP-4616.txt
        12 kB
        Pete Wyckoff
      4. HADOOP-4616_0.18.2.txt
        6 kB
        Pete Wyckoff
      5. HADOOP-4616_0.18.2.txt
        6 kB
        Pete Wyckoff
      6. HADOOP-4616_0.19.txt
        7 kB
        Pete Wyckoff
      7. HADOOP-4616_0.19.txt
        7 kB
        Pete Wyckoff

        Issue Links

          Activity

            People

            • Assignee:
              Pete Wyckoff
              Reporter:
              Marc-Olivier Fleury
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development