Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4616

assertion makes fuse-dfs exit when reading incomplete data

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.2
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When trying to read a file that is corrupt on HDFS (registered by the namenode, but part of the data is missing on the datanodes), some of the assertions in dfs_read fail, causing the program to abort. This makes it impossible to access the mounted partition until it is mounted again.

      A simple way to reproduce this bug is to remove enough datanodes to have part of the data missing, and to read each file listed in HDFS.

      this is the assertion that fails (fuse_dfs.c:903) : assert(bufferReadIndex >= 0 && bufferReadIndex < fh->bufferSize);

      The expected behaviour would be to return either no file or a corrupt file, but continue working afterward.

      removing the assertion seems to work for now, but a special behaviour is probably needed to handle this particular problem correctly.

        Attachments

        1. HADOOP-4616.txt
          2 kB
          Pete Wyckoff
        2. HADOOP-4616.txt
          2 kB
          Pete Wyckoff
        3. HADOOP-4616.txt
          12 kB
          Pete Wyckoff
        4. HADOOP-4616_0.19.txt
          7 kB
          Pete Wyckoff
        5. HADOOP-4616_0.19.txt
          7 kB
          Pete Wyckoff
        6. HADOOP-4616_0.18.2.txt
          6 kB
          Pete Wyckoff
        7. HADOOP-4616_0.18.2.txt
          6 kB
          Pete Wyckoff

          Issue Links

            Activity

              People

              • Assignee:
                wyckoff Pete Wyckoff
                Reporter:
                mofleury Marc-Olivier Fleury
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: