Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-200

In HDFS, sync() not yet guarantees data available to the new readers

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 0.20-append
    • 0.20-append, 0.20.205.0
    • None
    • None
    • hbase

    Description

      In the append design doc (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it says

      • A reader is guaranteed to be able to read data that was 'flushed' before the reader opened the file

      However, this feature is not yet implemented. Note that the operation 'flushed' is now called "sync".

      Attachments

        1. 4379_20081010TC3.java
          4 kB
          Tsz-wo Sze
        2. fsyncConcurrentReaders.txt
          14 kB
          Dhruba Borthakur
        3. Writer.java
          2 kB
          Jim Kellerman
        4. Reader.java
          1 kB
          Jim Kellerman
        5. fsyncConcurrentReaders3.patch
          14 kB
          Dhruba Borthakur
        6. Writer.java
          2 kB
          Dhruba Borthakur
        7. Reader.java
          2 kB
          Dhruba Borthakur
        8. fsyncConcurrentReaders4.patch
          15 kB
          Dhruba Borthakur
        9. hypertable-namenode.log.gz
          33 kB
          Doug Judd
        10. ReopenProblem.java
          7 kB
          Doug Judd
        11. reopen_test.sh
          0.3 kB
          Doug Judd
        12. namenode.log
          95 kB
          Jim Kellerman
        13. namenode.log
          2.51 MB
          Jim Kellerman
        14. fsyncConcurrentReaders5.txt
          15 kB
          Dhruba Borthakur
        15. fsyncConcurrentReaders6.patch
          15 kB
          Dhruba Borthakur
        16. fsyncConcurrentReaders9.patch
          19 kB
          Dhruba Borthakur
        17. fsyncConcurrentReaders11_20.txt
          14 kB
          Dhruba Borthakur
        18. hadoop-stack-namenode-aa0-000-12.u.powerset.com.log.gz
          86 kB
          Michael Stack
        19. fsyncConcurrentReaders12_20.txt
          14 kB
          Dhruba Borthakur
        20. fsyncConcurrentReaders13_20.txt
          15 kB
          Dhruba Borthakur
        21. fsyncConcurrentReaders14_20.txt
          17 kB
          Dhruba Borthakur
        22. hdfs-200-ryan-existing-file-fail.txt
          13 kB
          ryan rawson
        23. fsyncConcurrentReaders15_20.txt
          18 kB
          Dhruba Borthakur
        24. fsyncConcurrentReaders16_20.txt
          18 kB
          Dhruba Borthakur
        25. checkLeases-fix-1.txt
          0.7 kB
          sam rash
        26. checkLeases-fix-unit-test-1.txt
          4 kB
          sam rash
        27. HDFS-200.20-security.1.patch
          18 kB
          Jitendra Nath Pandey

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-495 Hadoop FSNamesystem startFileInternal() getLease() has bug

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HBASE-1148 Always flush HLog on root or meta region updates

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-1155 Verify that FSDataoutputStream.sync() works

          • Major - Major loss of function.
          • Closed
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1779 After NameNode restart , Clients can not read partial files even after client invokes Sync.

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-376 FileSystem.listStatus should report the current length of a file if it has been sync'd

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue HDFS-659 DFSClient cannot read all the available bytes

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-158 HDFS soft lease limit should be configurable per-file

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-2658 Design and Implement a Test Plan to support appends to HDFS files

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1700 Append to files in HDFS

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-630 In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.

          • Major - Major loss of function.
          • Closed

          Activity

            People

              dhruba Dhruba Borthakur
              szetszwo Tsz-wo Sze
              Votes:
              8 Vote for this issue
              Watchers:
              40 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: