Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3110

libhdfs implementation of direct read API

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: libhdfs, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      libhdfs is enhanced to read directly into user-supplied buffers when possible, reducing the number of memory copies.

      Description

      Once HDFS-2834 gets committed, we can add support for the new API to libhdfs, which leads to significant performance increases when reading local data from C.

      1. HDFS-3110.0.patch
        4 kB
        Henry Robinson
      2. HDFS-3110.1.patch
        13 kB
        Henry Robinson
      3. HDFS-3110.2.patch
        20 kB
        Henry Robinson
      4. HDFS-3110.3.patch
        19 kB
        Henry Robinson
      5. HDFS-3110.4.patch
        20 kB
        Henry Robinson
      6. HDFS-3110.5.patch
        20 kB
        Henry Robinson

        Issue Links

          Activity

          Henry Robinson created issue -
          Henry Robinson made changes -
          Field Original Value New Value
          Link This issue depends on HDFS-2834 [ HDFS-2834 ]
          Henry Robinson made changes -
          Component/s libhdfs [ 12313126 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.0.patch [ 12519573 ]
          Henry Robinson made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.1.patch [ 12520868 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.2.patch [ 12521180 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.3.patch [ 12521235 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.4.patch [ 12521262 ]
          Henry Robinson made changes -
          Attachment HDFS-3110.5.patch [ 12521434 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Release Note libhdfs is enhanced to read directly into user-supplied buffers when possible, reducing the number of memory copies.
          Todd Lipcon made changes -
          Component/s performance [ 12316501 ]
          Todd Lipcon made changes -
          Fix Version/s 2.0.1-alpha [ 12321440 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322472 ]
          Fix Version/s 2.1.0-alpha [ 12321440 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends on HDFS-2834 [ HDFS-2834 ]
          Gavin made changes -
          Link This issue depends upon HDFS-2834 [ HDFS-2834 ]

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development