Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1, 1.0.0, 0.22.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
    1. 0001-HDFS-347.-Local-reads.patch
      48 kB
      Andrew Purtell
    2. HDFS-2246.20s.1.patch
      48 kB
      Suresh Srinivas
    3. HDFS-2246.20s.2.txt
      53 kB
      Suresh Srinivas
    4. HDFS-2246.20s.3.txt
      53 kB
      Suresh Srinivas
    5. HDFS-2246.20s.4.txt
      55 kB
      Jitendra Nath Pandey
    6. HDFS-2246.20s.patch
      46 kB
      Suresh Srinivas
    7. HDFS-2246-22.patch
      118 kB
      Benoy Antony
    8. HDFS-2246-branch-0.20-security.3.patch
      65 kB
      Jitendra Nath Pandey
    9. HDFS-2246-branch-0.20-security.no-softref.patch
      62 kB
      Jitendra Nath Pandey
    10. HDFS-2246-branch-0.20-security.patch
      65 kB
      Jitendra Nath Pandey
    11. HDFS-2246-branch-0.20-security.patch
      63 kB
      Jitendra Nath Pandey
    12. HDFS-2246-branch-0.20-security.patch
      61 kB
      Jitendra Nath Pandey
    13. HDFS-2246-branch-0.20-security.patch
      59 kB
      Jitendra Nath Pandey
    14. HDFS-2246-branch-0.20-security-205.1.patch
      60 kB
      Jitendra Nath Pandey
    15. HDFS-2246-branch-0.20-security-205.2.patch
      61 kB
      Jitendra Nath Pandey
    16. HDFS-2246-branch-0.20-security-205.patch
      62 kB
      Jitendra Nath Pandey
    17. HDFS-2246-branch-0.20-security-205.patch
      59 kB
      Jitendra Nath Pandey
    18. HDFS-2246-branch-0.20-security-205.patch
      208 kB
      Jitendra Nath Pandey
    19. HDFS-2246-trunk.patch
      69 kB
      Jitendra Nath Pandey
    20. HDFS-2246-trunk.patch
      69 kB
      Jitendra Nath Pandey
    21. HDFS-2246-trunk.patch
      68 kB
      Jitendra Nath Pandey
    22. HDFS-2246-trunk.patch
      68 kB
      Jitendra Nath Pandey
    23. HDFS-2246-trunk.patch
      60 kB
      Jitendra Nath Pandey
    24. HDFS-2246-trunk.patch
      60 kB
      Jitendra Nath Pandey
    25. localReadShortcut20-security.2patch
      47 kB
      Sanjay Radia
    26. TestShortCircuitLocalRead.java
      10 kB
      Todd Lipcon

      Issue Links

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Sanjay Radia
          • Votes:
            0 Vote for this issue
            Watchers:
            45 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development