Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1, 1.0.0, 0.22.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.

      Attachments

      1. HDFS-2246-22.patch
        118 kB
        Benoy Antony
      2. HDFS-2246-branch-0.20-security.patch
        65 kB
        Jitendra Nath Pandey
      3. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      4. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      5. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      6. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      7. HDFS-2246-branch-0.20-security.patch
        63 kB
        Jitendra Nath Pandey
      8. TestShortCircuitLocalRead.java
        10 kB
        Todd Lipcon
      9. HDFS-2246-branch-0.20-security-205.2.patch
        61 kB
        Jitendra Nath Pandey
      10. HDFS-2246-branch-0.20-security.patch
        61 kB
        Jitendra Nath Pandey
      11. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      12. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      13. HDFS-2246-branch-0.20-security-205.1.patch
        60 kB
        Jitendra Nath Pandey
      14. HDFS-2246-branch-0.20-security-205.patch
        62 kB
        Jitendra Nath Pandey
      15. HDFS-2246-branch-0.20-security.no-softref.patch
        62 kB
        Jitendra Nath Pandey
      16. HDFS-2246-branch-0.20-security.3.patch
        65 kB
        Jitendra Nath Pandey
      17. HDFS-2246-branch-0.20-security-205.patch
        59 kB
        Jitendra Nath Pandey
      18. HDFS-2246-branch-0.20-security-205.patch
        208 kB
        Jitendra Nath Pandey
      19. HDFS-2246-branch-0.20-security.patch
        59 kB
        Jitendra Nath Pandey
      20. HDFS-2246.20s.4.txt
        55 kB
        Jitendra Nath Pandey
      21. HDFS-2246.20s.3.txt
        53 kB
        Suresh Srinivas
      22. HDFS-2246.20s.2.txt
        53 kB
        Suresh Srinivas
      23. HDFS-2246.20s.1.patch
        48 kB
        Suresh Srinivas
      24. HDFS-2246.20s.patch
        46 kB
        Suresh Srinivas
      25. localReadShortcut20-security.2patch
        47 kB
        Sanjay Radia
      26. 0001-HDFS-347.-Local-reads.patch
        48 kB
        Andrew Purtell

        Issue Links

          Activity

            People

            • Assignee:
              jnp Jitendra Nath Pandey
              Reporter:
              sanjay.radia Sanjay Radia
            • Votes:
              0 Vote for this issue
              Watchers:
              45 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: