Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1060 Append/flush should support concurrent "tailer" use case
  3. HDFS-1057

Concurrent readers hit ChecksumExceptions if following a writer to very end of file

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before calling flush(). Therefore, if there is a concurrent reader, it's possible to race here - the reader will see the new length while those bytes are still in the buffers of BlockReceiver. Thus the client will potentially see checksum errors or EOFs. Additionally, the last checksum chunk of the file is made accessible to readers even though it is not stable.

      1. hdfs-1057-trunk-6.txt
        29 kB
        sam rash
      2. hdfs-1057-trunk-5.txt
        30 kB
        sam rash
      3. hdfs-1057-trunk-4.txt
        29 kB
        sam rash
      4. hdfs-1057-trunk-3.txt
        25 kB
        sam rash
      5. hdfs-1057-trunk-2.txt
        26 kB
        sam rash
      6. hdfs-1057-trunk-1.txt
        27 kB
        sam rash
      7. HDFS-1057-0.20-append.patch
        35 kB
        Nicolas Spiegelberg
      8. HDFS-1057.20-security.1.patch
        34 kB
        Jitendra Nath Pandey
      9. conurrent-reader-patch-3.txt
        34 kB
        sam rash
      10. conurrent-reader-patch-2.txt
        31 kB
        sam rash
      11. conurrent-reader-patch-1.txt
        30 kB
        sam rash

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              sam rash
              Reporter:
              Todd Lipcon
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development