Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1, 1.0.0, 0.22.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.

      Attachments

      1. 0001-HDFS-347.-Local-reads.patch
        48 kB
        Andrew Purtell
      2. HDFS-2246.20s.1.patch
        48 kB
        Suresh Srinivas
      3. HDFS-2246.20s.2.txt
        53 kB
        Suresh Srinivas
      4. HDFS-2246.20s.3.txt
        53 kB
        Suresh Srinivas
      5. HDFS-2246.20s.4.txt
        55 kB
        Jitendra Nath Pandey
      6. HDFS-2246.20s.patch
        46 kB
        Suresh Srinivas
      7. HDFS-2246-22.patch
        118 kB
        Benoy Antony
      8. HDFS-2246-branch-0.20-security.3.patch
        65 kB
        Jitendra Nath Pandey
      9. HDFS-2246-branch-0.20-security.no-softref.patch
        62 kB
        Jitendra Nath Pandey
      10. HDFS-2246-branch-0.20-security.patch
        65 kB
        Jitendra Nath Pandey
      11. HDFS-2246-branch-0.20-security.patch
        63 kB
        Jitendra Nath Pandey
      12. HDFS-2246-branch-0.20-security.patch
        61 kB
        Jitendra Nath Pandey
      13. HDFS-2246-branch-0.20-security.patch
        59 kB
        Jitendra Nath Pandey
      14. HDFS-2246-branch-0.20-security-205.1.patch
        60 kB
        Jitendra Nath Pandey
      15. HDFS-2246-branch-0.20-security-205.2.patch
        61 kB
        Jitendra Nath Pandey
      16. HDFS-2246-branch-0.20-security-205.patch
        62 kB
        Jitendra Nath Pandey
      17. HDFS-2246-branch-0.20-security-205.patch
        59 kB
        Jitendra Nath Pandey
      18. HDFS-2246-branch-0.20-security-205.patch
        208 kB
        Jitendra Nath Pandey
      19. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      20. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      21. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      22. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      23. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      24. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      25. localReadShortcut20-security.2patch
        47 kB
        Sanjay Radia
      26. TestShortCircuitLocalRead.java
        10 kB
        Todd Lipcon

        Issue Links

          Activity

            People

            • Assignee:
              jnp Jitendra Nath Pandey
              Reporter:
              sanjay.radia Sanjay Radia
            • Votes:
              0 Vote for this issue
              Watchers:
              45 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: