Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4616

assertion makes fuse-dfs exit when reading incomplete data

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.18.2
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When trying to read a file that is corrupt on HDFS (registered by the namenode, but part of the data is missing on the datanodes), some of the assertions in dfs_read fail, causing the program to abort. This makes it impossible to access the mounted partition until it is mounted again.

      A simple way to reproduce this bug is to remove enough datanodes to have part of the data missing, and to read each file listed in HDFS.

      this is the assertion that fails (fuse_dfs.c:903) : assert(bufferReadIndex >= 0 && bufferReadIndex < fh->bufferSize);

      The expected behaviour would be to return either no file or a corrupt file, but continue working afterward.

      removing the assertion seems to work for now, but a special behaviour is probably needed to handle this particular problem correctly.

        Attachments

        1. HADOOP-4616.txt
          2 kB
          Pete Wyckoff
        2. HADOOP-4616.txt
          2 kB
          Pete Wyckoff
        3. HADOOP-4616.txt
          12 kB
          Pete Wyckoff
        4. HADOOP-4616_0.18.2.txt
          6 kB
          Pete Wyckoff
        5. HADOOP-4616_0.18.2.txt
          6 kB
          Pete Wyckoff
        6. HADOOP-4616_0.19.txt
          7 kB
          Pete Wyckoff
        7. HADOOP-4616_0.19.txt
          7 kB
          Pete Wyckoff

        Issue Links

          Activity

            People

            • Assignee:
              wyckoff Pete Wyckoff
              Reporter:
              mofleury Marc-Olivier Fleury

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment