Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1700

Append to files in HDFS

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.15.1
    • 0.19.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Introduced append operation for HDFS files.
      Show
      Introduced append operation for HDFS files.

    Description

      Request for being able to append to files in HDFS has been raised a couple of times on the list of late. For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193. Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480 (Later on this thread, Jim Kellerman re-raises the HBase need of this feature). HADOOP-337 'DFS files should be appendable' makes mention of file append but it was opened early in the life of HDFS when the focus was more on implementing the basics rather than adding new features. Interest fizzled. Because HADOOP-337 is also a bit of a grab-bag – it includes truncation and being able to concurrently read/write – rather than try and breathe new life into HADOOP-337, instead, here is a new issue focused on file append. Ultimately, being able to do as the google GFS paper describes – having multiple concurrent clients making 'Atomic Record Append' to a single file would be sweet but at least for a first cut at this feature, IMO, a single client appending to a single HDFS file letting the application manage the access would be sufficent.

      Attachments

        1. appendtrunk16.patch
          80 kB
          Dhruba Borthakur
        2. appendtrunk15.patch
          80 kB
          Dhruba Borthakur
        3. appendtrunk14.patch
          79 kB
          Dhruba Borthakur
        4. appendtrunk14.patch
          80 kB
          Dhruba Borthakur
        5. appendtrunk13.patch
          79 kB
          Dhruba Borthakur
        6. appendtrunk13.patch
          79 kB
          Dhruba Borthakur
        7. appendtrunk13.patch
          64 kB
          Dhruba Borthakur
        8. appendtrunk12.patch
          80 kB
          Tsz-wo Sze
        9. appendtrunk11.patch
          80 kB
          Tsz-wo Sze
        10. appendtrunk10.patch
          81 kB
          Tsz-wo Sze
        11. appendtrunk9.patch
          79 kB
          Tsz-wo Sze
        12. appendtrunk8.patch
          76 kB
          Tsz-wo Sze
        13. appendtrunk7.patch
          73 kB
          Dhruba Borthakur
        14. appendtrunk6.patch
          73 kB
          Dhruba Borthakur
        15. append3.patch
          55 kB
          Dhruba Borthakur
        16. 1700_20080606.patch
          14 kB
          Tsz-wo Sze
        17. Grid_HadoopRenumberBlocks.pdf
          74 kB
          Robert Chansler
        18. append.patch
          22 kB
          Ruyue Ma
        19. Appends.doc
          76 kB
          Dhruba Borthakur
        20. Appends.html
          45 kB
          Dhruba Borthakur
        21. Appends.doc
          70 kB
          Dhruba Borthakur

        Issue Links

          Activity

            People

              dhruba Dhruba Borthakur
              stack Michael Stack
              Votes:
              11 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: