Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.1
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Introduced append operation for HDFS files.
      Show
      Introduced append operation for HDFS files.

      Description

      Request for being able to append to files in HDFS has been raised a couple of times on the list of late. For one example, see http://www.nabble.com/HDFS%2C-appending-writes-status-tf3848237.html#a10916193. Other mail describes folks' workarounds because this feature is lacking: e.g. http://www.nabble.com/Loading-data-into-HDFS-tf4200003.html#a12039480 (Later on this thread, Jim Kellerman re-raises the HBase need of this feature). HADOOP-337 'DFS files should be appendable' makes mention of file append but it was opened early in the life of HDFS when the focus was more on implementing the basics rather than adding new features. Interest fizzled. Because HADOOP-337 is also a bit of a grab-bag – it includes truncation and being able to concurrently read/write – rather than try and breathe new life into HADOOP-337, instead, here is a new issue focused on file append. Ultimately, being able to do as the google GFS paper describes – having multiple concurrent clients making 'Atomic Record Append' to a single file would be sweet but at least for a first cut at this feature, IMO, a single client appending to a single HDFS file letting the application manage the access would be sufficent.

      1. Appends.doc
        70 kB
        dhruba borthakur
      2. Appends.html
        45 kB
        dhruba borthakur
      3. Appends.doc
        76 kB
        dhruba borthakur
      4. append.patch
        22 kB
        Ruyue Ma
      5. Grid_HadoopRenumberBlocks.pdf
        74 kB
        Robert Chansler
      6. 1700_20080606.patch
        14 kB
        Tsz Wo Nicholas Sze
      7. append3.patch
        55 kB
        dhruba borthakur
      8. appendtrunk6.patch
        73 kB
        dhruba borthakur
      9. appendtrunk7.patch
        73 kB
        dhruba borthakur
      10. appendtrunk8.patch
        76 kB
        Tsz Wo Nicholas Sze
      11. appendtrunk9.patch
        79 kB
        Tsz Wo Nicholas Sze
      12. appendtrunk10.patch
        81 kB
        Tsz Wo Nicholas Sze
      13. appendtrunk11.patch
        80 kB
        Tsz Wo Nicholas Sze
      14. appendtrunk12.patch
        80 kB
        Tsz Wo Nicholas Sze
      15. appendtrunk13.patch
        64 kB
        dhruba borthakur
      16. appendtrunk13.patch
        79 kB
        dhruba borthakur
      17. appendtrunk13.patch
        79 kB
        dhruba borthakur
      18. appendtrunk14.patch
        80 kB
        dhruba borthakur
      19. appendtrunk14.patch
        79 kB
        dhruba borthakur
      20. appendtrunk15.patch
        80 kB
        dhruba borthakur
      21. appendtrunk16.patch
        80 kB
        dhruba borthakur

        Issue Links

          Activity

          Konstantin Shvachko made changes -
          Link This issue is superceded by HDFS-265 [ HDFS-265 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-3283 [ HADOOP-3283 ]
          Gavin made changes -
          Link This issue depends on HADOOP-3283 [ HADOOP-3283 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-3310 [ HADOOP-3310 ]
          Gavin made changes -
          Link This issue depends on HADOOP-3310 [ HADOOP-3310 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Release Note The protocol used by the DFSClient to Datanode and Namenode has been modified.
          A new HadoopMetric named "FilesAppended" is added. This metric shows the number of times any file was opened for appends.
          Introduced append operation for HDFS files.
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HADOOP-4379 [ HADOOP-4379 ]
          dhruba borthakur made changes -
          Link This issue relates to HADOOP-3834 [ HADOOP-3834 ]
          dhruba borthakur made changes -
          Link This issue relates to HADOOP-3833 [ HADOOP-3833 ]
          dhruba borthakur made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          dhruba borthakur made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Fix Version/s 0.19.0 [ 12313211 ]
          dhruba borthakur made changes -
          Link This issue relates to HADOOP-3832 [ HADOOP-3832 ]
          dhruba borthakur made changes -
          Release Note The protocol used by the DFSClient to Datanode and Namenode has been modified.
          A new HadoopMetric named "FilesAppended" is added. This metric shows the number of times any file was opened for appends.
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed] [Incompatible change, Reviewed]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          dhruba borthakur made changes -
          Attachment appendtrunk16.patch [ 12386795 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Attachment appendtrunk15.patch [ 12386453 ]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          dhruba borthakur made changes -
          Link This issue is depended upon by HADOOP-3790 [ HADOOP-3790 ]
          dhruba borthakur made changes -
          Attachment appendtrunk14.patch [ 12386249 ]
          dhruba borthakur made changes -
          Attachment appendtrunk14.patch [ 12386245 ]
          dhruba borthakur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Attachment appendtrunk13.patch [ 12386234 ]
          dhruba borthakur made changes -
          Attachment appendtrunk13.patch [ 12386222 ]
          dhruba borthakur made changes -
          Attachment appendtrunk13.patch [ 12386122 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment appendtrunk12.patch [ 12386089 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment appendtrunk11.patch [ 12386082 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment appendtrunk10.patch [ 12386029 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment appendtrunk9.patch [ 12386013 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment appendtrunk8.patch [ 12385696 ]
          dhruba borthakur made changes -
          Attachment appendtrunk7.patch [ 12385544 ]
          dhruba borthakur made changes -
          Attachment appendtrunk6.patch [ 12385448 ]
          dhruba borthakur made changes -
          Assignee dhruba borthakur [ dhruba ]
          dhruba borthakur made changes -
          Attachment append3.patch [ 12384904 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3515 [ HADOOP-3515 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 1700_20080606.patch [ 12383601 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3503 [ HADOOP-3503 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3177 [ HADOOP-3177 ]
          Robert Chansler made changes -
          Attachment Grid_HadoopRenumberBlocks.pdf [ 12381639 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-3329 [ HADOOP-3329 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3177 [ HADOOP-3177 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue depends on HADOOP-3310 [ HADOOP-3310 ]
          Doug Cutting made changes -
          Link This issue is blocked by HADOOP-2565 [ HADOOP-2565 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue depends on HADOOP-3283 [ HADOOP-3283 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3113 [ HADOOP-3113 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3250 [ HADOOP-3250 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-3241 [ HADOOP-3241 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3201 [ HADOOP-3201 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3177 [ HADOOP-3177 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3176 [ HADOOP-3176 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-3161 [ HADOOP-3161 ]
          dhruba borthakur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-2658 [ HADOOP-2658 ]
          dhruba borthakur made changes -
          Link This issue relates to HADOOP-2657 [ HADOOP-2657 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-2656 [ HADOOP-2656 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-2655 [ HADOOP-2655 ]
          Ruyue Ma made changes -
          Affects Version/s 0.15.1 [ 12312850 ]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ruyue Ma made changes -
          Attachment append.patch [ 12372198 ]
          Robert Chansler made changes -
          Link This issue is related to HADOOP-337 [ HADOOP-337 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-2345 [ HADOOP-2345 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12370562 ]
          dhruba borthakur made changes -
          Attachment Appends.htm [ 12367999 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12369579 ]
          dhruba borthakur made changes -
          Attachment Appends-1.xhtml [ 12368024 ]
          dhruba borthakur made changes -
          Attachment Appends.html [ 12369639 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12369638 ]
          dhruba borthakur made changes -
          Link This issue relates to HADOOP-1497 [ HADOOP-1497 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12368000 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12369579 ]
          Doug Cutting made changes -
          Attachment Appends-1.xhtml [ 12368024 ]
          dhruba borthakur made changes -
          Link This issue is blocked by HADOOP-1707 [ HADOOP-1707 ]
          dhruba borthakur made changes -
          Attachment Appends.doc [ 12368000 ]
          dhruba borthakur made changes -
          Attachment Appends.htm [ 12367999 ]
          dhruba borthakur made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-89 [ HADOOP-89 ]
          stack created issue -

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              stack
            • Votes:
              11 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development