Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1707

Remove the DFS Client disk-based cache

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.16.0
    • None
    • None

    Description

      The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.

      A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.

      Attachments

        1. clientDiskBuffer27.patch
          136 kB
          Dhruba Borthakur
        2. clientDiskBuffer27.patch
          136 kB
          Dhruba Borthakur
        3. clientDiskBuffer26.patch
          136 kB
          Dhruba Borthakur
        4. clientDiskBuffer25.patch
          135 kB
          Dhruba Borthakur
        5. clientDiskBuffer24.patch
          132 kB
          Dhruba Borthakur
        6. clientDiskBuffer24.patch
          116 kB
          Dhruba Borthakur
        7. clientDiskBuffer23.patch
          113 kB
          Dhruba Borthakur
        8. clientDiskBuffer23.patch
          113 kB
          Dhruba Borthakur
        9. clientDiskBuffer22.patch
          129 kB
          Dhruba Borthakur
        10. clientDiskBuffer21.patch
          129 kB
          Dhruba Borthakur
        11. clientDiskBuffer20.patch
          129 kB
          Dhruba Borthakur
        12. clientDiskBuffer19.patch
          126 kB
          Dhruba Borthakur
        13. clientDiskBuffer18.patch
          125 kB
          Dhruba Borthakur
        14. clientDiskBuffer17.patch
          126 kB
          Dhruba Borthakur
        15. clientDiskBuffer16.patch
          117 kB
          Dhruba Borthakur
        16. clientDiskBuffer15.patch
          111 kB
          Dhruba Borthakur
        17. clientDiskBuffer14.patch
          99 kB
          Dhruba Borthakur
        18. clientDiskBuffer12.patch
          97 kB
          Dhruba Borthakur
        19. clientDiskBuffer11.patch
          97 kB
          Dhruba Borthakur
        20. clientDiskBuffer10.patch
          97 kB
          Dhruba Borthakur
        21. clientDiskBuffer9.patch
          97 kB
          Dhruba Borthakur
        22. clientDiskBuffer8.patch
          96 kB
          Dhruba Borthakur
        23. clientDiskBuffer7.patch
          94 kB
          Dhruba Borthakur
        24. DataTransferProtocol.html
          18 kB
          Dhruba Borthakur
        25. DataTransferProtocol.doc
          44 kB
          Dhruba Borthakur
        26. clientDiskBuffer6.patch
          89 kB
          Dhruba Borthakur
        27. clientDiskBuffer2.patch
          49 kB
          Dhruba Borthakur
        28. clientDiskBuffer.patch
          45 kB
          Dhruba Borthakur

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-1700 Append to files in HDFS

          • Major - Major loss of function.
          • Closed
          depends upon

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-1927 a datanode in a write pipeline should report an error if the next datanode in the pipeline reported an error

          • Major - Major loss of function.
          • Closed
          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-67 /tmp/hadoop-${user}/dfs/tmp/tmp/client-${long}.tmp is not cleanup correctly

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-445 Parallel data/socket writing for DFSOutputStream

          • Major - Major loss of function.
          • Closed

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: