Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1707

Remove the DFS Client disk-based cache

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.

      A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.

        Attachments

        1. clientDiskBuffer.patch
          45 kB
          dhruba borthakur
        2. clientDiskBuffer2.patch
          49 kB
          dhruba borthakur
        3. clientDiskBuffer6.patch
          89 kB
          dhruba borthakur
        4. DataTransferProtocol.doc
          44 kB
          dhruba borthakur
        5. DataTransferProtocol.html
          18 kB
          dhruba borthakur
        6. clientDiskBuffer7.patch
          94 kB
          dhruba borthakur
        7. clientDiskBuffer8.patch
          96 kB
          dhruba borthakur
        8. clientDiskBuffer9.patch
          97 kB
          dhruba borthakur
        9. clientDiskBuffer10.patch
          97 kB
          dhruba borthakur
        10. clientDiskBuffer11.patch
          97 kB
          dhruba borthakur
        11. clientDiskBuffer12.patch
          97 kB
          dhruba borthakur
        12. clientDiskBuffer14.patch
          99 kB
          dhruba borthakur
        13. clientDiskBuffer15.patch
          111 kB
          dhruba borthakur
        14. clientDiskBuffer16.patch
          117 kB
          dhruba borthakur
        15. clientDiskBuffer17.patch
          126 kB
          dhruba borthakur
        16. clientDiskBuffer18.patch
          125 kB
          dhruba borthakur
        17. clientDiskBuffer19.patch
          126 kB
          dhruba borthakur
        18. clientDiskBuffer20.patch
          129 kB
          dhruba borthakur
        19. clientDiskBuffer21.patch
          129 kB
          dhruba borthakur
        20. clientDiskBuffer22.patch
          129 kB
          dhruba borthakur
        21. clientDiskBuffer23.patch
          113 kB
          dhruba borthakur
        22. clientDiskBuffer23.patch
          113 kB
          dhruba borthakur
        23. clientDiskBuffer24.patch
          116 kB
          dhruba borthakur
        24. clientDiskBuffer24.patch
          132 kB
          dhruba borthakur
        25. clientDiskBuffer25.patch
          135 kB
          dhruba borthakur
        26. clientDiskBuffer26.patch
          136 kB
          dhruba borthakur
        27. clientDiskBuffer27.patch
          136 kB
          dhruba borthakur
        28. clientDiskBuffer27.patch
          136 kB
          dhruba borthakur

          Issue Links

            Activity

              People

              • Assignee:
                dhruba dhruba borthakur
                Reporter:
                dhruba dhruba borthakur
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: