Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1707

Remove the DFS Client disk-based cache

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.16.0
    • None
    • None

    Description

      The DFS client currently uses a staging file on local disk to cache all user-writes to a file. When the staging file accumulates 1 block worth of data, its contents are flushed to a HDFS datanode. These operations occur sequentially.

      A simple optimization of allowing the user to write to another staging file while simultaneously uploading the contents of the first staging file to HDFS will improve file-upload performance.

      Attachments

        1. DataTransferProtocol.html
          18 kB
          Dhruba Borthakur
        2. DataTransferProtocol.doc
          44 kB
          Dhruba Borthakur
        3. clientDiskBuffer9.patch
          97 kB
          Dhruba Borthakur
        4. clientDiskBuffer8.patch
          96 kB
          Dhruba Borthakur
        5. clientDiskBuffer7.patch
          94 kB
          Dhruba Borthakur
        6. clientDiskBuffer6.patch
          89 kB
          Dhruba Borthakur
        7. clientDiskBuffer27.patch
          136 kB
          Dhruba Borthakur
        8. clientDiskBuffer27.patch
          136 kB
          Dhruba Borthakur
        9. clientDiskBuffer26.patch
          136 kB
          Dhruba Borthakur
        10. clientDiskBuffer25.patch
          135 kB
          Dhruba Borthakur
        11. clientDiskBuffer24.patch
          116 kB
          Dhruba Borthakur
        12. clientDiskBuffer24.patch
          132 kB
          Dhruba Borthakur
        13. clientDiskBuffer23.patch
          113 kB
          Dhruba Borthakur
        14. clientDiskBuffer23.patch
          113 kB
          Dhruba Borthakur
        15. clientDiskBuffer22.patch
          129 kB
          Dhruba Borthakur
        16. clientDiskBuffer21.patch
          129 kB
          Dhruba Borthakur
        17. clientDiskBuffer20.patch
          129 kB
          Dhruba Borthakur
        18. clientDiskBuffer2.patch
          49 kB
          Dhruba Borthakur
        19. clientDiskBuffer19.patch
          126 kB
          Dhruba Borthakur
        20. clientDiskBuffer18.patch
          125 kB
          Dhruba Borthakur
        21. clientDiskBuffer17.patch
          126 kB
          Dhruba Borthakur
        22. clientDiskBuffer16.patch
          117 kB
          Dhruba Borthakur
        23. clientDiskBuffer15.patch
          111 kB
          Dhruba Borthakur
        24. clientDiskBuffer14.patch
          99 kB
          Dhruba Borthakur
        25. clientDiskBuffer12.patch
          97 kB
          Dhruba Borthakur
        26. clientDiskBuffer11.patch
          97 kB
          Dhruba Borthakur
        27. clientDiskBuffer10.patch
          97 kB
          Dhruba Borthakur
        28. clientDiskBuffer.patch
          45 kB
          Dhruba Borthakur

        Issue Links

          Activity

            gmcdonald Gavin McDonald made changes -
            Link This issue depends upon HADOOP-1927 [ HADOOP-1927 ]
            gmcdonald Gavin McDonald made changes -
            Link This issue depends on HADOOP-1927 [ HADOOP-1927 ]
            omalley Owen O'Malley made changes -
            Component/s dfs [ 12310710 ]
            nidaley Nigel Daley made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            dhruba Dhruba Borthakur made changes -
            Resolution Fixed [ 1 ]
            Status Patch Available [ 10002 ] Resolved [ 5 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer27.patch [ 12373344 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer27.patch [ 12373337 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer26.patch [ 12373299 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            shv Konstantin Shvachko made changes -
            Link This issue incorporates HADOOP-2561 [ HADOOP-2561 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer25.patch [ 12373225 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer24.patch [ 12373189 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer24.patch [ 12373147 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer23.patch [ 12373138 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer23.patch [ 12373133 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer22.patch [ 12373117 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer21.patch [ 12373081 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer20.patch [ 12373048 ]
            dhruba Dhruba Borthakur made changes -
            Status Patch Available [ 10002 ] Open [ 1 ]
            dhruba Dhruba Borthakur made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer19.patch [ 12372949 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer18.patch [ 12372728 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer17.patch [ 12372646 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer16.patch [ 12372326 ]
            chansler Robert Chansler made changes -
            Link This issue is related to HADOOP-445 [ HADOOP-445 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer15.patch [ 12371931 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer14.patch [ 12371568 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer12.patch [ 12371209 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer11.patch [ 12371080 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer10.patch [ 12371005 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer9.patch [ 12370884 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer8.patch [ 12370340 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer7.patch [ 12369962 ]
            dhruba Dhruba Borthakur made changes -
            Attachment DataTransferProtocol.html [ 12369626 ]
            dhruba Dhruba Borthakur made changes -
            Attachment DataTransferProtocol.doc [ 12369625 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer6.patch [ 12369417 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer2.patch [ 12368891 ]
            dhruba Dhruba Borthakur made changes -
            Attachment clientDiskBuffer.patch [ 12368789 ]
            dhruba Dhruba Borthakur made changes -
            Link This issue blocks HADOOP-1700 [ HADOOP-1700 ]
            dhruba Dhruba Borthakur made changes -
            Link This issue depends on HADOOP-1927 [ HADOOP-1927 ]
            dhruba Dhruba Borthakur made changes -
            Fix Version/s 0.16.0 [ 12312740 ]
            Summary DFS client can allow user to write data to the next block while uploading previous block to HDFS Remove the DFS Client disk-based cache
            dhruba Dhruba Borthakur made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            dhruba Dhruba Borthakur made changes -
            Field Original Value New Value
            Assignee dhruba borthakur [ dhruba ]
            dhruba Dhruba Borthakur created issue -

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: