Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1060 Append/flush should support concurrent "tailer" use case
  3. HDFS-1057

Concurrent readers hit ChecksumExceptions if following a writer to very end of file

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before calling flush(). Therefore, if there is a concurrent reader, it's possible to race here - the reader will see the new length while those bytes are still in the buffers of BlockReceiver. Thus the client will potentially see checksum errors or EOFs. Additionally, the last checksum chunk of the file is made accessible to readers even though it is not stable.

        Attachments

        1. conurrent-reader-patch-1.txt
          30 kB
          sam rash
        2. conurrent-reader-patch-2.txt
          31 kB
          sam rash
        3. conurrent-reader-patch-3.txt
          34 kB
          sam rash
        4. hdfs-1057-trunk-1.txt
          27 kB
          sam rash
        5. hdfs-1057-trunk-2.txt
          26 kB
          sam rash
        6. hdfs-1057-trunk-3.txt
          25 kB
          sam rash
        7. hdfs-1057-trunk-4.txt
          29 kB
          sam rash
        8. HDFS-1057-0.20-append.patch
          35 kB
          Nicolas Spiegelberg
        9. hdfs-1057-trunk-5.txt
          30 kB
          sam rash
        10. hdfs-1057-trunk-6.txt
          29 kB
          sam rash
        11. HDFS-1057.20-security.1.patch
          34 kB
          Jitendra Nath Pandey

          Issue Links

            Activity

              People

              • Assignee:
                rash37 sam rash
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                1 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: