Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1060 Append/flush should support concurrent "tailer" use case
  3. HDFS-1057

Concurrent readers hit ChecksumExceptions if following a writer to very end of file

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before calling flush(). Therefore, if there is a concurrent reader, it's possible to race here - the reader will see the new length while those bytes are still in the buffers of BlockReceiver. Thus the client will potentially see checksum errors or EOFs. Additionally, the last checksum chunk of the file is made accessible to readers even though it is not stable.

      Attachments

        1. conurrent-reader-patch-1.txt
          30 kB
          sam rash
        2. conurrent-reader-patch-2.txt
          31 kB
          sam rash
        3. conurrent-reader-patch-3.txt
          34 kB
          sam rash
        4. hdfs-1057-trunk-1.txt
          27 kB
          sam rash
        5. hdfs-1057-trunk-2.txt
          26 kB
          sam rash
        6. hdfs-1057-trunk-3.txt
          25 kB
          sam rash
        7. hdfs-1057-trunk-4.txt
          29 kB
          sam rash
        8. HDFS-1057-0.20-append.patch
          35 kB
          Nicolas Spiegelberg
        9. hdfs-1057-trunk-5.txt
          30 kB
          sam rash
        10. hdfs-1057-trunk-6.txt
          29 kB
          sam rash
        11. HDFS-1057.20-security.1.patch
          34 kB
          Jitendra Nath Pandey

        Issue Links

          Activity

            People

              rash37 sam rash
              tlipcon Todd Lipcon
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: