Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.23.1, 1.0.0, 0.22.1
    • None
    • None
    • Reviewed
    • Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.

    Attachments

      1. TestShortCircuitLocalRead.java
        10 kB
        Todd Lipcon
      2. localReadShortcut20-security.2patch
        47 kB
        Sanjay Radia
      3. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      4. HDFS-2246-trunk.patch
        60 kB
        Jitendra Nath Pandey
      5. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      6. HDFS-2246-trunk.patch
        68 kB
        Jitendra Nath Pandey
      7. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      8. HDFS-2246-trunk.patch
        69 kB
        Jitendra Nath Pandey
      9. HDFS-2246-branch-0.20-security-205.patch
        208 kB
        Jitendra Nath Pandey
      10. HDFS-2246-branch-0.20-security-205.patch
        59 kB
        Jitendra Nath Pandey
      11. HDFS-2246-branch-0.20-security-205.patch
        62 kB
        Jitendra Nath Pandey
      12. HDFS-2246-branch-0.20-security-205.2.patch
        61 kB
        Jitendra Nath Pandey
      13. HDFS-2246-branch-0.20-security-205.1.patch
        60 kB
        Jitendra Nath Pandey
      14. HDFS-2246-branch-0.20-security.patch
        59 kB
        Jitendra Nath Pandey
      15. HDFS-2246-branch-0.20-security.patch
        61 kB
        Jitendra Nath Pandey
      16. HDFS-2246-branch-0.20-security.patch
        63 kB
        Jitendra Nath Pandey
      17. HDFS-2246-branch-0.20-security.patch
        65 kB
        Jitendra Nath Pandey
      18. HDFS-2246-branch-0.20-security.no-softref.patch
        62 kB
        Jitendra Nath Pandey
      19. HDFS-2246-branch-0.20-security.3.patch
        65 kB
        Jitendra Nath Pandey
      20. HDFS-2246-22.patch
        118 kB
        Benoy Antony
      21. HDFS-2246.20s.patch
        46 kB
        Suresh Srinivas
      22. HDFS-2246.20s.4.txt
        55 kB
        Jitendra Nath Pandey
      23. HDFS-2246.20s.3.txt
        53 kB
        Suresh Srinivas
      24. HDFS-2246.20s.2.txt
        53 kB
        Suresh Srinivas
      25. HDFS-2246.20s.1.patch
        48 kB
        Suresh Srinivas
      26. 0001-HDFS-347.-Local-reads.patch
        48 kB
        Andrew Kyle Purtell

      Issue Links

      duplicates

      Improvement - An improvement or enhancement to an existing feature or task. HDFS-281 Explore usage of the sendfile api via java.nio.channels.FileChannel.transfer{To|From} for i/o in datanodes

      • Major - Major loss of function.
      • Resolved
      is related to

      Improvement - An improvement or enhancement to an existing feature or task. HADOOP-7804 enable hadoop config generator to set dfs.block.local-path-access.user to enable short circuit read

      • Major - Major loss of function.
      • Closed

      Improvement - An improvement or enhancement to an existing feature or task. HDFS-3907 Allow multiple users for local block readers

      • Major - Major loss of function.
      • Closed
      relates to

      New Feature - A new feature of the product, which has yet to be developed. HADOOP-6311 Add support for unix domain sockets to JNI libs

      • Major - Major loss of function.
      • Resolved

      New Feature - A new feature of the product, which has yet to be developed. HDFS-245 Create symbolic links in HDFS

      • Major - Major loss of function.
      • Closed

      Improvement - An improvement or enhancement to an existing feature or task. HDFS-347 DFS read performance suboptimal when client co-located on nodes with data

      • Major - Major loss of function.
      • Closed

      Improvement - An improvement or enhancement to an existing feature or task. HDFS-2654 Make BlockReaderLocal not extend RemoteBlockReader2

      • Major - Major loss of function.
      • Closed

      Activity
        This comment will be Viewable by All Users Viewable by All Users
        Cancel

        People

          jnp Jitendra Nath Pandey
          sanjay.radia Sanjay Radia
          Votes:
          0 Vote for this issue
          Watchers:
          44 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved:

            Slack

              Issue deployment