Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2246

Shortcut a local client reads to a Datanodes files directly

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1, 1.0.0, 0.22.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1. New configurations
      a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
      b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
      c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
      2. By default none of the above are enabled and short circuit read will not kick in.
      3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
      Show
      1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
    • Target Version/s:
    1. HDFS-2246-22.patch
      118 kB
      Benoy Antony
    2. HDFS-2246-branch-0.20-security.patch
      65 kB
      Jitendra Nath Pandey
    3. HDFS-2246-trunk.patch
      69 kB
      Jitendra Nath Pandey
    4. HDFS-2246-trunk.patch
      69 kB
      Jitendra Nath Pandey
    5. HDFS-2246-trunk.patch
      68 kB
      Jitendra Nath Pandey
    6. HDFS-2246-trunk.patch
      68 kB
      Jitendra Nath Pandey
    7. HDFS-2246-branch-0.20-security.patch
      63 kB
      Jitendra Nath Pandey
    8. TestShortCircuitLocalRead.java
      10 kB
      Todd Lipcon
    9. HDFS-2246-branch-0.20-security-205.2.patch
      61 kB
      Jitendra Nath Pandey
    10. HDFS-2246-branch-0.20-security.patch
      61 kB
      Jitendra Nath Pandey
    11. HDFS-2246-trunk.patch
      60 kB
      Jitendra Nath Pandey
    12. HDFS-2246-trunk.patch
      60 kB
      Jitendra Nath Pandey
    13. HDFS-2246-branch-0.20-security-205.1.patch
      60 kB
      Jitendra Nath Pandey
    14. HDFS-2246-branch-0.20-security-205.patch
      62 kB
      Jitendra Nath Pandey
    15. HDFS-2246-branch-0.20-security.no-softref.patch
      62 kB
      Jitendra Nath Pandey
    16. HDFS-2246-branch-0.20-security.3.patch
      65 kB
      Jitendra Nath Pandey
    17. HDFS-2246-branch-0.20-security-205.patch
      59 kB
      Jitendra Nath Pandey
    18. HDFS-2246-branch-0.20-security-205.patch
      208 kB
      Jitendra Nath Pandey
    19. HDFS-2246-branch-0.20-security.patch
      59 kB
      Jitendra Nath Pandey
    20. HDFS-2246.20s.4.txt
      55 kB
      Jitendra Nath Pandey
    21. HDFS-2246.20s.3.txt
      53 kB
      Suresh Srinivas
    22. HDFS-2246.20s.2.txt
      53 kB
      Suresh Srinivas
    23. HDFS-2246.20s.1.patch
      48 kB
      Suresh Srinivas
    24. HDFS-2246.20s.patch
      46 kB
      Suresh Srinivas
    25. localReadShortcut20-security.2patch
      47 kB
      Sanjay Radia
    26. 0001-HDFS-347.-Local-reads.patch
      48 kB
      Andrew Purtell

      Issue Links

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #129 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/129/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh, Jitendra and Benoy (Revision 1329468)

        Result = FAILURE
        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329468
        Files :

        • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/BlockReader.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/RemoteBlockReader.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/commit-tests
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataXceiver.java
        • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockTokenWithDFS.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #129 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/129/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh, Jitendra and Benoy (Revision 1329468) Result = FAILURE shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329468 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/BlockReader.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/RemoteBlockReader.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java /hadoop/common/branches/branch-0.22/hdfs/src/test/commit-tests /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/BlockReaderTestUtil.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestClientBlockVerification.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataXceiver.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockTokenWithDFS.java
        Hide
        Konstantin Shvachko added a comment -

        Committed to branch 0.22.1. Thanks Benoy for porting.

        Show
        Konstantin Shvachko added a comment - Committed to branch 0.22.1. Thanks Benoy for porting.
        Hide
        Konstantin Shvachko added a comment -

        +1 the patch looks good.
        Passes all tests and it ran in internal branch for some time.

        Show
        Konstantin Shvachko added a comment - +1 the patch looks good. Passes all tests and it ran in internal branch for some time.
        Hide
        Benoy Antony added a comment -

        The attached patch for 22 is ported from patch for 205.

        Show
        Benoy Antony added a comment - The attached patch for 22 is ported from patch for 205.
        Hide
        Matt Foley added a comment -

        Closed upon release of version 1.0.0.

        Show
        Matt Foley added a comment - Closed upon release of version 1.0.0.
        Hide
        Eli Collins added a comment -

        This could because the patch I generated for JD is 2246 plus 2654 (which is not in 1.0). I wouldn't be surprised if there's some side-effect due to the local reader extending the remote block reader in 205.

        Show
        Eli Collins added a comment - This could because the patch I generated for JD is 2246 plus 2654 (which is not in 1.0). I wouldn't be surprised if there's some side-effect due to the local reader extending the remote block reader in 205.
        Hide
        Todd Lipcon added a comment -

        Also the new checksum stuff is in 0.23 but I think JD is testing the 1.0.0 (nee 0.20.205) patch. It's a mystery!

        Show
        Todd Lipcon added a comment - Also the new checksum stuff is in 0.23 but I think JD is testing the 1.0.0 (nee 0.20.205) patch. It's a mystery!
        Hide
        Jean-Daniel Cryans added a comment -

        If you're on nehalem Todd's work to make the checksumming much faster is kicking in.

        I am, but it's when it's not checksumming that u3 is much much faster than 1.0 else with checksumming if there's a difference it's minimal.

        Show
        Jean-Daniel Cryans added a comment - If you're on nehalem Todd's work to make the checksumming much faster is kicking in. I am, but it's when it's not checksumming that u3 is much much faster than 1.0 else with checksumming if there's a difference it's minimal.
        Hide
        dhruba borthakur added a comment -

        Hi JD, I agree that this is a scenario that I did not think of ahead. Funny! I guess it should not cause a problem though...

        Show
        dhruba borthakur added a comment - Hi JD, I agree that this is a scenario that I did not think of ahead. Funny! I guess it should not cause a problem though...
        Hide
        Eli Collins added a comment -

        If you're on nehalem Todd's work to make the checksumming much faster is kicking in.

        Show
        Eli Collins added a comment - If you're on nehalem Todd's work to make the checksumming much faster is kicking in.
        Hide
        Jean-Daniel Cryans added a comment -

        Something else I've seen is that disabling checksumming has almost no effect, but when trying the patch for cdh3u3 I do get a >2x speedup (and I think my bottleneck isn't even from reading now). What's weird is that I can see that it's not doing the crc32s in the former case, but still almost the same speed.

        Show
        Jean-Daniel Cryans added a comment - Something else I've seen is that disabling checksumming has almost no effect, but when trying the patch for cdh3u3 I do get a >2x speedup (and I think my bottleneck isn't even from reading now). What's weird is that I can see that it's not doing the crc32s in the former case, but still almost the same speed.
        Hide
        Jean-Daniel Cryans added a comment -

        Testing local reads, I encountered an interesting problem. If you kill -9 the a DN that has local data to a client (like a region server), the client will continue using the local blocks. It seems like a change in semantics to me as in normally if you kill a DN you can't read from it, but now you can if it's local.

        I agree that it's a rare case, the DN is dead but not the machine nor its clients, and I can't think of a problem caused by this a the moment.

        Show
        Jean-Daniel Cryans added a comment - Testing local reads, I encountered an interesting problem. If you kill -9 the a DN that has local data to a client (like a region server), the client will continue using the local blocks. It seems like a change in semantics to me as in normally if you kill a DN you can't read from it, but now you can if it's local. I agree that it's a rare case, the DN is dead but not the machine nor its clients, and I can't think of a problem caused by this a the moment.
        Hide
        Matt Foley added a comment -

        corrected Fix Versions field

        Show
        Matt Foley added a comment - corrected Fix Versions field
        Hide
        Jitendra Nath Pandey added a comment -

        Committed to 206,205 as well.

        Show
        Jitendra Nath Pandey added a comment - Committed to 206,205 as well.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #103 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/103/)
        Merged r1204792 from trunk for HDFS-2246.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #103 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/103/ ) Merged r1204792 from trunk for HDFS-2246 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #86 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/86/)
        Merged r1204792 from trunk for HDFS-2246.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #86 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/86/ ) Merged r1204792 from trunk for HDFS-2246 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #209 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/209/)
        Merged r1204792 from trunk for HDFS-2246.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #209 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/209/ ) Merged r1204792 from trunk for HDFS-2246 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #212 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/212/)
        Merged r1204792 from trunk for HDFS-2246.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #212 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/212/ ) Merged r1204792 from trunk for HDFS-2246 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #223 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/223/)
        Merged r1204792 from trunk for HDFS-2246.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #223 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/223/ ) Merged r1204792 from trunk for HDFS-2246 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205694 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Jitendra Nath Pandey added a comment -

        Committed to 206.

        Show
        Jitendra Nath Pandey added a comment - Committed to 206.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the 206 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the 206 patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504820/HDFS-2246-branch-0.20-security.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1606//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504820/HDFS-2246-branch-0.20-security.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1606//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Patch for 206, uploaded

        Show
        Jitendra Nath Pandey added a comment - Patch for 206, uploaded
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #905 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #905 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/905/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #871 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/871/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #871 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/871/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1310 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1310/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1310 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1310/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1383 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1383/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1383 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1383/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1333 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1333/)
        HDFS-2246. Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1333 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1333/ ) HDFS-2246 . Enable reading a block directly from local file system for a client on the same node as the block file. Contributed by Andrew Purtell, Suresh Srinivas and Jitendra Nath Pandey szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1204792 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolServerSideTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeProtocolTranslatorR23.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ClientDatanodeWireProtocol.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the latest trunk patch looks good.

        I have committed it. Thanks Andrew, Suresh and Jitendra!

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the latest trunk patch looks good. I have committed it. Thanks Andrew, Suresh and Jitendra!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504683/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 8 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.viewfs.TestViewFsHdfs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1602//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1602//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504683/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsHdfs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1602//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1602//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch to fix merge issue.

        Show
        Jitendra Nath Pandey added a comment - Updated patch to fix merge issue.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504676/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 8 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1601//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504676/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1601//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Addressed Nic's comments.

        Show
        Jitendra Nath Pandey added a comment - Addressed Nic's comments.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Harsh, the key is supposed to be dfs.client.read.shortcircuit.skip.checksum. It does not sound right to rename it to dfs.client.read.shortcircuit.skip.checksum.enabled.

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Harsh, the key is supposed to be dfs.client.read.shortcircuit. skip .checksum. It does not sound right to rename it to dfs.client.read.shortcircuit. skip.checksum.enabled .
        Hide
        Harsh J added a comment -

        Could boolean-valued config names be suffixed with a .enabled for clarity?

        Show
        Harsh J added a comment - Could boolean-valued config names be suffixed with a .enabled for clarity?
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • The value should also be updated
          +  public static final String DFS_CLIENT_READ_SKIP_CHECKSUM_KEY = "dfs.client.read.shortcircuit.checksum";
          
        • Some public methods do not have javadoc.
        Show
        Tsz Wo Nicholas Sze added a comment - The value should also be updated + public static final String DFS_CLIENT_READ_SKIP_CHECKSUM_KEY = "dfs.client.read.shortcircuit.checksum" ; Some public methods do not have javadoc.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504531/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 8 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.fs.viewfs.TestViewFsHdfs

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1597//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1597//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504531/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsHdfs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1597//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1597//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch fixing the merge issue.

        Show
        Jitendra Nath Pandey added a comment - Updated patch fixing the merge issue.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504416/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 8 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1587//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504416/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1587//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Alright, here is an updated trunk patch which addresses most of Todd's comments and also incorporates his updated benchmark. Following comments I would prefer to address as followup items: Refactor BlockReaderLocal and not extend RemoteBlockReader2, Exploring ACL to enable this feature for multiple users and key the cache with UGI.

        Show
        Jitendra Nath Pandey added a comment - Alright, here is an updated trunk patch which addresses most of Todd's comments and also incorporates his updated benchmark. Following comments I would prefer to address as followup items: Refactor BlockReaderLocal and not extend RemoteBlockReader2, Exploring ACL to enable this feature for multiple users and key the cache with UGI.
        Hide
        Todd Lipcon added a comment -

        I agree with Eli. What's the point in defining a policy around maintenance releases if we break it so quickly? I'm -1 on a patch going into 0.20 until there's one in trunk, and I think the initial code reviews should be happening on trunk, followed by a backport.

        Show
        Todd Lipcon added a comment - I agree with Eli. What's the point in defining a policy around maintenance releases if we break it so quickly? I'm -1 on a patch going into 0.20 until there's one in trunk, and I think the initial code reviews should be happening on trunk, followed by a backport.
        Hide
        Eli Collins added a comment -

        What 205.1 deadline are you referring to? According to Matt the code freeze for 205.1 was last Friday. Per the long discussion on list (and all the bugs Todd found with the patch after it was reviewed) this isn't an appropriate change for a dot release of a dot release, especially at such a late date in the release. How about we let this stabilize in 206 first?

        Show
        Eli Collins added a comment - What 205.1 deadline are you referring to? According to Matt the code freeze for 205.1 was last Friday. Per the long discussion on list (and all the bugs Todd found with the patch after it was reviewed) this isn't an appropriate change for a dot release of a dot release, especially at such a late date in the release. How about we let this stabilize in 206 first?
        Hide
        Jitendra Nath Pandey added a comment -

        Todd,
        I am working on the trunk patch, but its a little bit bigger change than 20s because of the very different implementation of BlockReaderLocal in trunk. Can we review/commit the 20S patch before that, because we are a bit under pressure on 205.1 deadline?

        Show
        Jitendra Nath Pandey added a comment - Todd, I am working on the trunk patch, but its a little bit bigger change than 20s because of the very different implementation of BlockReaderLocal in trunk. Can we review/commit the 20S patch before that, because we are a bit under pressure on 205.1 deadline?
        Hide
        Todd Lipcon added a comment -

        Is a new version of the trunk patch coming as well? I was figuring I'd review trunk patches as they showed up, and then review a backport patch once the trunk one was final.

        Show
        Todd Lipcon added a comment - Is a new version of the trunk patch coming as well? I was figuring I'd review trunk patches as they showed up, and then review a backport patch once the trunk one was final.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > Is it ok if I address the ACL configuration and ugi in cache keys in a separate jira for trunk and let 20s support only one user?

        This makes sense.

        One minor comment for the latest patch:

        • The name "dfs.client.read.shortcircuit.checksum" is not clear whether it is to enable checksum or it is to skip checksum.

        Todd, would you like review the 0.20 patch?

        Show
        Tsz Wo Nicholas Sze added a comment - > Is it ok if I address the ACL configuration and ugi in cache keys in a separate jira for trunk and let 20s support only one user? This makes sense. One minor comment for the latest patch: The name "dfs.client.read.shortcircuit.checksum" is not clear whether it is to enable checksum or it is to skip checksum. Todd, would you like review the 0.20 patch?
        Hide
        Jitendra Nath Pandey added a comment -

        Rather than a single user for the local path access, why not use an ACL, given we already have support for parsing/checking ACLs? I can imagine some people may have multiple applications that do local random-read access that could benefit from this.

        The ACLs are configured for a protocol so we need to add a protocol just for getBlockLocalPathInfo, which seems a bit overkill because the protocol will be too specific to this special optimization.

        The cached proxies to the local DN currently don't have the UGI as part of the key. So if a client needs to impersonate different users, the cache will end up caching client proxies associated with previous users, which would be incorrect.

        In this patch short circuit can't be used with impersonation because we check for authentication method as KERBEROS at the datanode. In case of impersonation the auth-method would be PROXY.
        However, we will have to do that once we enable short circuit for multiple users.

        Is it ok if I address the ACL configuration and ugi in cache keys in a separate jira for trunk and let 20s support only one user?

        Show
        Jitendra Nath Pandey added a comment - Rather than a single user for the local path access, why not use an ACL, given we already have support for parsing/checking ACLs? I can imagine some people may have multiple applications that do local random-read access that could benefit from this. The ACLs are configured for a protocol so we need to add a protocol just for getBlockLocalPathInfo, which seems a bit overkill because the protocol will be too specific to this special optimization. The cached proxies to the local DN currently don't have the UGI as part of the key. So if a client needs to impersonate different users, the cache will end up caching client proxies associated with previous users, which would be incorrect. In this patch short circuit can't be used with impersonation because we check for authentication method as KERBEROS at the datanode. In case of impersonation the auth-method would be PROXY. However, we will have to do that once we enable short circuit for multiple users. Is it ok if I address the ACL configuration and ugi in cache keys in a separate jira for trunk and let 20s support only one user?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504007/HDFS-2246-branch-0.20-security.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1563//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504007/HDFS-2246-branch-0.20-security.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1563//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch for branch-0.20-security. It fixes most of Todd's comments that are relevant to 20s except the ugi based cache and ACLs.

        Show
        Jitendra Nath Pandey added a comment - Updated patch for branch-0.20-security. It fixes most of Todd's comments that are relevant to 20s except the ugi based cache and ACLs.
        Hide
        Todd Lipcon added a comment -

        10 threads, with checksum on. I think the benefit would be bigger with a couple more optimizations (eg amortizing checksum checks across larger reads, caching the block header info, etc).

        Would be interesting to see if this carries over to TestParallelRead, which is a more random-IO test.

        Show
        Todd Lipcon added a comment - 10 threads, with checksum on. I think the benefit would be bigger with a couple more optimizations (eg amortizing checksum checks across larger reads, caching the block header info, etc). Would be interesting to see if this carries over to TestParallelRead, which is a more random-IO test.
        Hide
        Jitendra Nath Pandey added a comment -

        > with these benchmarks, the optimization seems to be about 3x vs trunk, which is pretty nice.
        That's awesome! How many threads you ran? Did you turn off the checksum?

        Show
        Jitendra Nath Pandey added a comment - > with these benchmarks, the optimization seems to be about 3x vs trunk, which is pretty nice. That's awesome! How many threads you ran? Did you turn off the checksum?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503687/TestShortCircuitLocalRead.java
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1558//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503687/TestShortCircuitLocalRead.java against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1558//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        attached fixed benchmark - I fixed the verification code to not spend all of its time in string concatenation, and fixed the random byte generation to only happen once at the top of the file. (50x speedup vs original benchmark).

        with these benchmarks, the optimization seems to be about 3x vs trunk, which is pretty nice.

        Show
        Todd Lipcon added a comment - attached fixed benchmark - I fixed the verification code to not spend all of its time in string concatenation, and fixed the random byte generation to only happen once at the top of the file. (50x speedup vs original benchmark). with these benchmarks, the optimization seems to be about 3x vs trunk, which is pretty nice.
        Hide
        Todd Lipcon added a comment -

        Another issue: if the client gets back the "not allowed to call getBlockLocalPathInfo" exception, it should disable the local-read path and continue without it. (otherwise you can't use the same hbase-site.xml or hdfs-site.xml between the servers and the clients)

        Show
        Todd Lipcon added a comment - Another issue: if the client gets back the "not allowed to call getBlockLocalPathInfo" exception, it should disable the local-read path and continue without it. (otherwise you can't use the same hbase-site.xml or hdfs-site.xml between the servers and the clients)
        Hide
        Todd Lipcon added a comment -

        another note: the "benchmark" included is in fact a benchmark of string formatting. All of the threads are spending the majority of their time at the following stack:

        "Thread-2" prio=10 tid=0x00007ff7b41ea800 nid=0x6502 runnable [0x00007ff7b3cfb000]
           java.lang.Thread.State: RUNNABLE
                at java.lang.Integer.getChars(Integer.java:370)
                at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:615)
                at java.lang.StringBuilder.append(StringBuilder.java:229)
                at org.apache.hadoop.hdfs.TestShortCircuitLocalRead.checkData(TestShortCircuitLocalRead.java:92)
                at org.apache.hadoop.hdfs.TestShortCircuitLocalRead.checkFullFile(TestShortCircuitLocalRead.java:118)
                at org.apache.hadoop.hdfs.TestShortCircuitLocalRead$3.run(TestShortCircuitLocalRead.java:269)
        
        Show
        Todd Lipcon added a comment - another note: the "benchmark" included is in fact a benchmark of string formatting. All of the threads are spending the majority of their time at the following stack: " Thread -2" prio=10 tid=0x00007ff7b41ea800 nid=0x6502 runnable [0x00007ff7b3cfb000] java.lang. Thread .State: RUNNABLE at java.lang. Integer .getChars( Integer .java:370) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:615) at java.lang.StringBuilder.append(StringBuilder.java:229) at org.apache.hadoop.hdfs.TestShortCircuitLocalRead.checkData(TestShortCircuitLocalRead.java:92) at org.apache.hadoop.hdfs.TestShortCircuitLocalRead.checkFullFile(TestShortCircuitLocalRead.java:118) at org.apache.hadoop.hdfs.TestShortCircuitLocalRead$3.run(TestShortCircuitLocalRead.java:269)
        Hide
        Todd Lipcon added a comment -

        btw, the new findbugs warnings in the test-patch results above seem to be truthful

        Show
        Todd Lipcon added a comment - btw, the new findbugs warnings in the test-patch results above seem to be truthful
        Hide
        Todd Lipcon added a comment -
        +  // Cache of block to file path where the block is stored
        +  //private static Map<ExtendedBlock, BlockLocalPathInfo> cache = null;
        

        remove dead code


        in LocalBlockReader.newBlockReader, if the checksum file open fails, then the dataIn stream will be leaked. You need a try..finally to close these if the open wasn't successful


        why does LocalBlockReader extend RemoteBlockReader2? It should be its own implementation of BlockReader - it doesn't really share much code with its superclass (looks like only skip, which is trivial anyway). In terms of member variables, it uses some, but ignores the rest.


        The cached proxies to the local DN currently don't have the UGI as part of the key. So if a client needs to impersonate different users, the cache will end up caching client proxies associated with previous users, which would be incorrect.

        Same is true of different timeouts, right?


        The implementation of read is fairly inefficient - reading and verifying checksum on one chunk at a time does a poor job of amortizing JNI/syscall overhead. Should probably read 64KB or 128KB at a time. You can address this in a followup.


        +            checksum.verifyChunkedSums(dataBuff, checksumBuff, filename, off);
        

        Wrong offset is passed here - you should be passing the position of dataIn (the block offset), not the offset into the target byte array


        The implementation of read looks incorrect to me. If you read less than one checksum chunk, and you want to verify checksums then it will always read an entire checksum chunk, so the next read will have incorrect results.


        In DFSClient, it looks very inefficient to be calling isLocalAddress without any caching.. iterating over the network interfaces is not cheap (lots of syscalls).

        Also, the way this is currently designed, each seek will result in the files being re-opened and the metadata re-parsed. I imagine this is bad for efficiency.


        Why not integrate the short circuit construction into getBlockReader instead of separately at the two call-sites?


        +import java.io.*;
        +
        +import org.apache.hadoop.io.*;
        

        please expand imports


        BlockLocalPathInfo needs audience/stability annotations. So does BlockMetadataHeader now that it's public.

        Also, since we now have the protocol-independence layer, does this .protocol class need to be Writable? Or can it be handled by the translation layer?


        Rather than a single user for the local path access, why not use an ACL, given we already have support for parsing/checking ACLs? I can imagine some people may have multiple applications that do local random-read access that could benefit from this.


        checkKerberosAuthMethod purports to be a general check, but the error message specifically references getBlockPathInfo.


        getBlockLocalPathInfo has some bad indentation. I'd also change those log messages to TRACE level.


        In FSDataset.getBlockLocalPathInfo, no need to qualify FSDataset.getMetaFile.


        Test wise, you should cover positional reads and differently sized reads. I think this would turn up a bug as pointed out above with reads smaller than a checksum chunk size.

        Show
        Todd Lipcon added a comment - + // Cache of block to file path where the block is stored + // private static Map<ExtendedBlock, BlockLocalPathInfo> cache = null ; remove dead code in LocalBlockReader.newBlockReader , if the checksum file open fails, then the dataIn stream will be leaked. You need a try..finally to close these if the open wasn't successful why does LocalBlockReader extend RemoteBlockReader2? It should be its own implementation of BlockReader - it doesn't really share much code with its superclass (looks like only skip , which is trivial anyway). In terms of member variables, it uses some, but ignores the rest. The cached proxies to the local DN currently don't have the UGI as part of the key. So if a client needs to impersonate different users, the cache will end up caching client proxies associated with previous users, which would be incorrect. Same is true of different timeouts, right? The implementation of read is fairly inefficient - reading and verifying checksum on one chunk at a time does a poor job of amortizing JNI/syscall overhead. Should probably read 64KB or 128KB at a time. You can address this in a followup. + checksum.verifyChunkedSums(dataBuff, checksumBuff, filename, off); Wrong offset is passed here - you should be passing the position of dataIn (the block offset), not the offset into the target byte array The implementation of read looks incorrect to me. If you read less than one checksum chunk, and you want to verify checksums then it will always read an entire checksum chunk, so the next read will have incorrect results. In DFSClient, it looks very inefficient to be calling isLocalAddress without any caching.. iterating over the network interfaces is not cheap (lots of syscalls). Also, the way this is currently designed, each seek will result in the files being re-opened and the metadata re-parsed. I imagine this is bad for efficiency. Why not integrate the short circuit construction into getBlockReader instead of separately at the two call-sites? + import java.io.*; + + import org.apache.hadoop.io.*; please expand imports BlockLocalPathInfo needs audience/stability annotations. So does BlockMetadataHeader now that it's public. Also, since we now have the protocol-independence layer, does this .protocol class need to be Writable? Or can it be handled by the translation layer? Rather than a single user for the local path access, why not use an ACL, given we already have support for parsing/checking ACLs? I can imagine some people may have multiple applications that do local random-read access that could benefit from this. checkKerberosAuthMethod purports to be a general check, but the error message specifically references getBlockPathInfo . getBlockLocalPathInfo has some bad indentation. I'd also change those log messages to TRACE level. In FSDataset.getBlockLocalPathInfo, no need to qualify FSDataset.getMetaFile . Test wise, you should cover positional reads and differently sized reads. I think this would turn up a bug as pointed out above with reads smaller than a checksum chunk size.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503476/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestAbandonBlock

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503476/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestAbandonBlock +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1556//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503475/HDFS-2246-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.TestDfsOverAvroRpc
        org.apache.hadoop.hdfs.TestFileAppend2
        org.apache.hadoop.hdfs.TestBalancerBandwidth
        org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
        org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503475/HDFS-2246-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1555//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12503479/HDFS-2246-branch-0.20-security-205.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1557//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503479/HDFS-2246-branch-0.20-security-205.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1557//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patches for trunk, 206 and 205.1 addressing Nic's comment.

        Show
        Jitendra Nath Pandey added a comment - Updated patches for trunk, 206 and 205.1 addressing Nic's comment.
        Hide
        Jitendra Nath Pandey added a comment -

        Patch for trunk uploaded.

        Show
        Jitendra Nath Pandey added a comment - Patch for trunk uploaded.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • In resetDatanodeProxy(), the if-condition should be null != proxy.

        Other than that the patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - In resetDatanodeProxy(), the if-condition should be null != proxy . Other than that the patch looks good.
        Hide
        Todd Lipcon added a comment -

        Can you please hold off until next week to commit this? Many of the HDFS developers were at ApacheCon and Hadoop World this week - I know I'd like a chance to review, but haven't yet.

        I'd also like to see a patch for trunk before this is released in a maintenance series. The policies on maintenance releases say that we should not be including new features until they're in trunk, as I understand them.

        Show
        Todd Lipcon added a comment - Can you please hold off until next week to commit this? Many of the HDFS developers were at ApacheCon and Hadoop World this week - I know I'd like a chance to review, but haven't yet. I'd also like to see a patch for trunk before this is released in a maintenance series. The policies on maintenance releases say that we should not be including new features until they're in trunk, as I understand them.
        Hide
        Jitendra Nath Pandey added a comment -

        Check DFS_CLIENT_READ_SHORTCIRCUIT when initializing userWithLocalPathAccess. What should happen if DFS_CLIENT_READ_SHORTCIRCUIT is false but DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY is set to some user?

        Datanode will allow getBlockLocalPathInfo if the user has local path access. DFS_CLIENT_READ_SHORTCIRCUIT is a client side configuration and should be configured in the application using the HDFS client.

        Show
        Jitendra Nath Pandey added a comment - Check DFS_CLIENT_READ_SHORTCIRCUIT when initializing userWithLocalPathAccess. What should happen if DFS_CLIENT_READ_SHORTCIRCUIT is false but DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY is set to some user? Datanode will allow getBlockLocalPathInfo if the user has local path access. DFS_CLIENT_READ_SHORTCIRCUIT is a client side configuration and should be configured in the application using the HDFS client.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch addressing nic's comments.

        Show
        Jitendra Nath Pandey added a comment - Updated patch addressing nic's comments.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • In BlockReaderLocal,
          • change newBlockReader(..) to package private
          • remove checkShortCircuitRead(..) since DFSClient has checked it.
          • remove initLocalDatanodeInfo(..); check null and init in getLocalDatanodeInfo(..)
          • add a local variable for LocalDatanodeInfo in getBlockPathInfo(..) and newBlockReader(..) so that we don't have to lookup the map multiple times.
          • Both pairs createDatanodeProxy/resetDatanodeProxy and getProxy/setProxy are synchronized. So one of the synchronizations is redundant. I suggest move createDatanodeProxy/resetDatanodeProxy to LocalDatanodeInfo and sync inside LocalDatanodeInfo.
        • change DFSClient.getLocalBlockReader(..) to private
        • In DataNode,
          • check DFS_CLIENT_READ_SHORTCIRCUIT when initializing userWithLocalPathAccess. What should happen if DFS_CLIENT_READ_SHORTCIRCUIT is false but DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY is set to some user?
        • use org.junit.Assert in TestShortCircuitLocalRead
        Show
        Tsz Wo Nicholas Sze added a comment - In BlockReaderLocal, change newBlockReader(..) to package private remove checkShortCircuitRead(..) since DFSClient has checked it. remove initLocalDatanodeInfo(..); check null and init in getLocalDatanodeInfo(..) add a local variable for LocalDatanodeInfo in getBlockPathInfo(..) and newBlockReader(..) so that we don't have to lookup the map multiple times. Both pairs createDatanodeProxy/resetDatanodeProxy and getProxy/setProxy are synchronized. So one of the synchronizations is redundant. I suggest move createDatanodeProxy/resetDatanodeProxy to LocalDatanodeInfo and sync inside LocalDatanodeInfo. change DFSClient.getLocalBlockReader(..) to private In DataNode, check DFS_CLIENT_READ_SHORTCIRCUIT when initializing userWithLocalPathAccess. What should happen if DFS_CLIENT_READ_SHORTCIRCUIT is false but DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY is set to some user? use org.junit.Assert in TestShortCircuitLocalRead
        Hide
        Matt Foley added a comment -

        Intent to patch is expressed in "Target Version", please mark "Fix Version" only after commit.

        Show
        Matt Foley added a comment - Intent to patch is expressed in "Target Version", please mark "Fix Version" only after commit.
        Hide
        Jitendra Nath Pandey added a comment -

        Patch for 205 branch.

        Show
        Jitendra Nath Pandey added a comment - Patch for 205 branch.
        Hide
        Jitendra Nath Pandey added a comment -

        Uploaded another patch with soft reference removed. Rest of it is identical to previous patch.

        Show
        Jitendra Nath Pandey added a comment - Uploaded another patch with soft reference removed. Rest of it is identical to previous patch.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch. Fix included to support multiple local datanodes. Addressed Nicholas's most of the comments, except removal of soft reference.

        Show
        Jitendra Nath Pandey added a comment - Updated patch. Fix included to support multiple local datanodes. Addressed Nicholas's most of the comments, except removal of soft reference.
        Hide
        Jitendra Nath Pandey added a comment -

        This jira was intended for 205 (as pointed out by Dhruba in an earlier comment) but couldn't be completed within 205 deadline, therefore we are now shooting for 205.1.

        Show
        Jitendra Nath Pandey added a comment - This jira was intended for 205 (as pointed out by Dhruba in an earlier comment) but couldn't be completed within 205 deadline, therefore we are now shooting for 205.1.
        Hide
        Todd Lipcon added a comment -

        I'm curious about the fixversion for this... it would seem 0.20.205.1 (a maintenance dot-release of a maintenance/stable branch) isn't exactly the place for a big new feature. What am I missing?

        Will try to review this soon, but it's a busy week for many of us at a conference.

        Show
        Todd Lipcon added a comment - I'm curious about the fixversion for this... it would seem 0.20.205.1 (a maintenance dot-release of a maintenance/stable branch) isn't exactly the place for a big new feature. What am I missing? Will try to review this soon, but it's a busy week for many of us at a conference.
        Hide
        Jitendra Nath Pandey added a comment -

        > synchronized LinkedHashMap

        I think LinkedHashMap has been used for a simple implementation of LRU policy.

        Show
        Jitendra Nath Pandey added a comment - > synchronized LinkedHashMap I think LinkedHashMap has been used for a simple implementation of LRU policy.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Comments on the client side:

        • In BlockReaderLocal,
          • In getBlockPathInfo(..), use the local variable proxy in the following. Otherwise, there is a race condition that datanodeProxy could be null.
                  pathinfo = datanodeProxy.getBlockLocalPathInfo(blk);
            
          • rename datanodeProxy to localhostProxy
          • replace the DatanodeInfo parameter newBlockReader with a port and use localhost:port to create proxy.
          • when should the proxy be stopped?
          • Why recordAllowedUser but not read the user from conf?
        Show
        Tsz Wo Nicholas Sze added a comment - Comments on the client side: In BlockReaderLocal, In getBlockPathInfo(..), use the local variable proxy in the following. Otherwise, there is a race condition that datanodeProxy could be null. pathinfo = datanodeProxy.getBlockLocalPathInfo(blk); rename datanodeProxy to localhostProxy replace the DatanodeInfo parameter newBlockReader with a port and use localhost:port to create proxy. when should the proxy be stopped? Why recordAllowedUser but not read the user from conf?
        Hide
        dhruba borthakur added a comment -

        > synchronized LinkedHashMap

        Why not a concurrent hash map? we have around 30K blocks per datanode and with SATA disks becoming 4TB or larger next year, we might have close to 100K blocks per datanode perhaps.

        Show
        dhruba borthakur added a comment - > synchronized LinkedHashMap Why not a concurrent hash map? we have around 30K blocks per datanode and with SATA disks becoming 4TB or larger next year, we might have close to 100K blocks per datanode perhaps.
        Hide
        Tsz Wo Nicholas Sze added a comment -
        • For DataNode.getBlockLocalPathInfo(..),
          • DataNode should check DFS_CLIENT_READ_SHORTCIRCUIT.
          • Should it calls checkBlockToken?
          • Should it checks whether the client address is local?
        • DataNode.getBlockLocalPathInfo(..) is not supposed to calls FSDataset.getMetaFile(..) directly. It should only calls FSDatasetInterface methods. How about change the new FSDatasetInterface.getBlockFile(..) to getBlockLocalPathInfo(..) wich returns BlockLocalPathInfo?
        • Change DataNode.userWithLocalPathAccess to final.
        • SoftLRUCache is implemented with SoftReference values. In general, it is a useful data structure. However, it may not be useful here since the size of the cache is about 10k and the size of value is small (< 1k). I suggest using a synchronized LinkedHashMap (no SoftReference) in BlockReaderLocal.cache and remove SoftLRUCache from the patch.
        • BlockReaderLocal.cache should not be initialized if checkShortCircuit is not enabled.

        I have not yet finished reading the client code.

        Show
        Tsz Wo Nicholas Sze added a comment - For DataNode.getBlockLocalPathInfo(..), DataNode should check DFS_CLIENT_READ_SHORTCIRCUIT. Should it calls checkBlockToken? Should it checks whether the client address is local? DataNode.getBlockLocalPathInfo(..) is not supposed to calls FSDataset.getMetaFile(..) directly. It should only calls FSDatasetInterface methods. How about change the new FSDatasetInterface.getBlockFile(..) to getBlockLocalPathInfo(..) wich returns BlockLocalPathInfo? Change DataNode.userWithLocalPathAccess to final. SoftLRUCache is implemented with SoftReference values. In general, it is a useful data structure. However, it may not be useful here since the size of the cache is about 10k and the size of value is small (< 1k). I suggest using a synchronized LinkedHashMap (no SoftReference) in BlockReaderLocal.cache and remove SoftLRUCache from the patch. BlockReaderLocal.cache should not be initialized if checkShortCircuit is not enabled. I have not yet finished reading the client code.
        Hide
        Jitendra Nath Pandey added a comment -

        Updated patch for 205 for latest state of the branch, addresses Arpit's comments.

        Show
        Jitendra Nath Pandey added a comment - Updated patch for 205 for latest state of the branch, addresses Arpit's comments.
        Hide
        Arpit Gupta added a comment -

        Also noticed that when running a hadoop cli saw the following being printed out..

        11/11/04 19:42:49 INFO hdfs.DFSClient: Short circuit read is false

        We should change the above to debug.

        Show
        Arpit Gupta added a comment - Also noticed that when running a hadoop cli saw the following being printed out.. 11/11/04 19:42:49 INFO hdfs.DFSClient: Short circuit read is false We should change the above to debug.
        Hide
        Arpit Gupta added a comment -

        the javadoc for BlockReaderLocal states

        The client performing short circuit reads must be running as the same user as that of the datanode.

        Where as the impl states that 'dfs.block.local-path-access.user' needs to set on the dn to the appropriate user. We should update the javadoc with that info.

        Show
        Arpit Gupta added a comment - the javadoc for BlockReaderLocal states The client performing short circuit reads must be running as the same user as that of the datanode. Where as the impl states that 'dfs.block.local-path-access.user' needs to set on the dn to the appropriate user. We should update the javadoc with that info.
        Hide
        Jitendra Nath Pandey added a comment -

        Patch for 205.

        Show
        Jitendra Nath Pandey added a comment - Patch for 205.
        Hide
        Jitendra Nath Pandey added a comment -

        The attached patch adds a server side configuration to specify the user who can use short circuit read.
        This parameter should be specified at the datanode. Datanode will allow getBlockLocalPathInfo only for this user.

        Show
        Jitendra Nath Pandey added a comment - The attached patch adds a server side configuration to specify the user who can use short circuit read. This parameter should be specified at the datanode. Datanode will allow getBlockLocalPathInfo only for this user.
        Hide
        dhruba borthakur added a comment -

        Hi Milind/Sanjay, I was assuming that this patch was targetted for hadoop-0.20.205.0, is that still true?

        Show
        dhruba borthakur added a comment - Hi Milind/Sanjay, I was assuming that this patch was targetted for hadoop-0.20.205.0, is that still true?
        Hide
        Milind Bhandarkar added a comment -

        @Sanjay, I used "container" too loosely. What I meant was that when run inside chrooted processes, the ip address remains the same, but the same path on dfs client and datanode refers to different things.

        Show
        Milind Bhandarkar added a comment - @Sanjay, I used "container" too loosely. What I meant was that when run inside chrooted processes, the ip address remains the same, but the same path on dfs client and datanode refers to different things.
        Hide
        Sanjay Radia added a comment -

        @milind ... we might [incorrectly] qualify a read for short circuit, on a multi container system ..

        In a multi-container environment such as linux containers or solaris zones, you are likely to give different IP addresses to each of the containers. Hence the problem you describe will not occur. Further, you would run HBase and DN within the same container or give each the same IP. I don't think linux container allow the sharing of IPs across containers.

        Show
        Sanjay Radia added a comment - @milind ... we might [incorrectly] qualify a read for short circuit, on a multi container system .. In a multi-container environment such as linux containers or solaris zones, you are likely to give different IP addresses to each of the containers. Hence the problem you describe will not occur. Further, you would run HBase and DN within the same container or give each the same IP. I don't think linux container allow the sharing of IPs across containers.
        Hide
        Suresh Srinivas added a comment -

        Got it. You mixing fd and HostComparator confused me. So HostComparator would work for this solution as well.

        Show
        Suresh Srinivas added a comment - Got it. You mixing fd and HostComparator confused me. So HostComparator would work for this solution as well.
        Hide
        Milind Bhandarkar added a comment -

        @Suresh, An example of an overridden host comparator would check if the same data-storage-specific file in / can be accessed by both datanode and dfsclient to determine if they are running on the same node. (Thus, shared storage and chroot issue can be addressed.)

        Show
        Milind Bhandarkar added a comment - @Suresh, An example of an overridden host comparator would check if the same data-storage-specific file in / can be accessed by both datanode and dfsclient to determine if they are running on the same node. (Thus, shared storage and chroot issue can be addressed.)
        Hide
        Suresh Srinivas added a comment -

        when a user-overridable method (HostComparator?) returns true

        can you tell me how it addresses the issues you raised?

        Show
        Suresh Srinivas added a comment - when a user-overridable method (HostComparator?) returns true can you tell me how it addresses the issues you raised?
        Hide
        Milind Bhandarkar added a comment -

        @Suresh, I think the right solution, (I recall that a similar suggestion was made earlier by Todd), is to share fd, when a user-overridable method (HostComparator?) returns true. (The default HostComparator will assume that ip addr equality means local filesystem namespace is identical.)

        Show
        Milind Bhandarkar added a comment - @Suresh, I think the right solution, (I recall that a similar suggestion was made earlier by Todd), is to share fd, when a user-overridable method (HostComparator?) returns true. (The default HostComparator will assume that ip addr equality means local filesystem namespace is identical.)
        Hide
        Suresh Srinivas added a comment -

        Reposting a comment that was dropped due to Jira reindexing.

        Comment from Milind on HDFS-7668:

        Suresh, I think you want a check to see if the same file representing a block is accessible from the client as well as the datanode. (Checking that both client and datanode is on the same host, is an overly restrictive check. In some ways, even if both client and datanode are run on the same node, with chrooted containers, this is an overly general check.) Right ?

        Currently based on the local host check, we might qualify a read for short circuit, on a multi container system. Short circuit read fails for such blocks and would eventually try reading the block over datanode, which succeeds. However the problem is, an unnecessary attempt to read locally is made, slowing down reads.

        I cannot think of a good solution. Should we compare the local hostname of the client with that of the hostname of the datanode, and allow the short circuit reads? The problem is, this might be too strict and may miss valid short circuit reads.

        Show
        Suresh Srinivas added a comment - Reposting a comment that was dropped due to Jira reindexing. Comment from Milind on HDFS-7668: Suresh, I think you want a check to see if the same file representing a block is accessible from the client as well as the datanode. (Checking that both client and datanode is on the same host, is an overly restrictive check. In some ways, even if both client and datanode are run on the same node, with chrooted containers, this is an overly general check.) Right ? Currently based on the local host check, we might qualify a read for short circuit, on a multi container system. Short circuit read fails for such blocks and would eventually try reading the block over datanode, which succeeds. However the problem is, an unnecessary attempt to read locally is made, slowing down reads. I cannot think of a good solution. Should we compare the local hostname of the client with that of the hostname of the datanode, and allow the short circuit reads? The problem is, this might be too strict and may miss valid short circuit reads.
        Hide
        Suresh Srinivas added a comment -

        After thinking a bit more about this, I realized the current solution is hard to use. Changing hbase to run from hbase user to hdfs user requires changing the ownership of all the HDFS files from hbase to hdfs. This is cumbersome.

        Another solution is to consider group based access.

        1. A group is created with hbase and hdfs in it
        2. A configuration option on HDFS is added for allowing short circuit access set to group and user for which it is allowed.
        3. When this config option is set, datanode during startup modifies storage directories and files read access to the configured group.
        4. Datanode#getPathInfo() allows the method call for the user configured in config option.
        5. Client still use the keberos authentication to maintain a single RPC proxy, as proposed earlier.
        Show
        Suresh Srinivas added a comment - After thinking a bit more about this, I realized the current solution is hard to use. Changing hbase to run from hbase user to hdfs user requires changing the ownership of all the HDFS files from hbase to hdfs. This is cumbersome. Another solution is to consider group based access. A group is created with hbase and hdfs in it A configuration option on HDFS is added for allowing short circuit access set to group and user for which it is allowed. When this config option is set, datanode during startup modifies storage directories and files read access to the configured group. Datanode#getPathInfo() allows the method call for the user configured in config option. Client still use the keberos authentication to maintain a single RPC proxy, as proposed earlier.
        Hide
        Suresh Srinivas added a comment -

        Passing the block token means we can still check block access based on HDFS permissions, and run as separate principals.

        The reason I removed this check is, the new method that is added currently is for short circuit reads only. It is allowed only for a user (same as datanode) and only over kerberos auth if security enabled. I like to start with such restrictions, making the API useful only for short circuit reads. Adding additional token based access is not adding much here and seems redundant.

        Also note my previous comment:

        getBlockPathInfo() is authorized using block token. This makes the use of cache for block to path info confusing. A cache could be added an entry based on one token and an access to the cache using another token finds the cached path info.

        Show
        Suresh Srinivas added a comment - Passing the block token means we can still check block access based on HDFS permissions, and run as separate principals. The reason I removed this check is, the new method that is added currently is for short circuit reads only. It is allowed only for a user (same as datanode) and only over kerberos auth if security enabled. I like to start with such restrictions, making the API useful only for short circuit reads. Adding additional token based access is not adding much here and seems redundant. Also note my previous comment: getBlockPathInfo() is authorized using block token. This makes the use of cache for block to path info confusing. A cache could be added an entry based on one token and an access to the cache using another token finds the cached path info.
        Hide
        Jitendra Nath Pandey added a comment -

        This patch is same as Suresh's patch with a little modification to add KerberosInfo annotation to ClientDataNodeProtocol. This is needed to make kerberos authenticated connection to datanode.

        Show
        Jitendra Nath Pandey added a comment - This patch is same as Suresh's patch with a little modification to add KerberosInfo annotation to ClientDataNodeProtocol. This is needed to make kerberos authenticated connection to datanode.
        Hide
        Todd Lipcon added a comment -

        ah, that makes some sense. Each proxy has a different token so they're cached separately at the IPC layer. We can call RPC.stopProxy but then we might run into HADOOP-7317.

        Would it be possible to use kerberos-authenticated RPC to the DN, and then pass the block token as a parameter to the method like was done in one of the earlier revs of the patch? It's a little "unorthodox" but gets around the connection caching issue. Passing the block token means we can still check block access based on HDFS permissions, and run as separate principals.

        Show
        Todd Lipcon added a comment - ah, that makes some sense. Each proxy has a different token so they're cached separately at the IPC layer. We can call RPC.stopProxy but then we might run into HADOOP-7317 . Would it be possible to use kerberos-authenticated RPC to the DN, and then pass the block token as a parameter to the method like was done in one of the earlier revs of the patch? It's a little "unorthodox" but gets around the connection caching issue. Passing the block token means we can still check block access based on HDFS permissions, and run as separate principals.
        Hide
        Suresh Srinivas added a comment -

        OOME I saw was related to creating a large number of RPC proxy and not related to the cache. I ran bunch of experiments:

        1. Create DN RPC proxy every time - this caused OOME (with or without cache). I did not dig deep into how many objects are created per RPC proxy.
        2. Tried disabling cache - this resulted in loss of perf gains.
        Show
        Suresh Srinivas added a comment - OOME I saw was related to creating a large number of RPC proxy and not related to the cache. I ran bunch of experiments: Create DN RPC proxy every time - this caused OOME (with or without cache). I did not dig deep into how many objects are created per RPC proxy. Tried disabling cache - this resulted in loss of perf gains.
        Hide
        Todd Lipcon added a comment -

        How do you assure relatively bounded number of blocks? Not sure what the HBase usecase here is

        Typically an HBase RS is responsible for <1TB or so of data. So likely to be bounded in the 10s of thousands of blocks on the high end. I'd imagine the memory usage for ~10K blocks would be pretty trivial (small number of MB). So, the OOME you saw is definitely concerning. Maybe we're better off using a bounded cache size instead of soft refs?

        Show
        Todd Lipcon added a comment - How do you assure relatively bounded number of blocks? Not sure what the HBase usecase here is Typically an HBase RS is responsible for <1TB or so of data. So likely to be bounded in the 10s of thousands of blocks on the high end. I'd imagine the memory usage for ~10K blocks would be pretty trivial (small number of MB). So, the OOME you saw is definitely concerning. Maybe we're better off using a bounded cache size instead of soft refs?
        Hide
        Todd Lipcon added a comment -

        On the other hand, running two different systems under the same security principal is also not a good idea. Good security practices are pretty antithetical to the way this patch works (hence my arguing that the proper solution with fd-passing should be finished instead of this hack)

        Most package-based installs of HBase do indeed run as a separate system user, and separate kerberos principals as well in secure HBase. I don't think doing access control based on a static assumption that the principal is the same is acceptable - perhaps an ACL is in order?

        Show
        Todd Lipcon added a comment - On the other hand, running two different systems under the same security principal is also not a good idea. Good security practices are pretty antithetical to the way this patch works (hence my arguing that the proper solution with fd-passing should be finished instead of this hack) Most package-based installs of HBase do indeed run as a separate system user, and separate kerberos principals as well in secure HBase. I don't think doing access control based on a static assumption that the principal is the same is acceptable - perhaps an ACL is in order?
        Hide
        Sanjay Radia added a comment -

        Using group access is generally not a good idea.
        In general a secure server should check that directories and files should be configured with the minimum access
        to run the the service securely. Hence the DN should check that DN user has RW access and that there is no group access or world access. Otherwise an accidental addition of member to the group would leave a security hole.

        Show
        Sanjay Radia added a comment - Using group access is generally not a good idea. In general a secure server should check that directories and files should be configured with the minimum access to run the the service securely. Hence the DN should check that DN user has RW access and that there is no group access or world access. Otherwise an accidental addition of member to the group would leave a security hole.
        Hide
        Suresh Srinivas added a comment -

        Could you explain that? I imagine the more likely deployment is that the client is running as 'hbase' and the DN is running as 'hdfs'. Then they would share a common group and have block files chmodded g+r.

        My understanding is that, both hbase region server and DN are running as the same user and not done using group access. Sanjay any comments?

        With the path cache, why's this true? It seems that, so long as you're accessing a relatively bounded number of blocks, the paths will all be cached and you won't need to re-RPC unless a block moves, etc?

        How do you assure relatively bounded number of blocks? Not sure what the HBase usecase here is. Additional information on the usage pattern would help. For every block to fill up the cache you end up creating an RPC proxy and a connection.

        The new patch is much cleaner. Also when I ran my client accessing 1000 blocks, I saw OutOfMemoryErrors.

        Show
        Suresh Srinivas added a comment - Could you explain that? I imagine the more likely deployment is that the client is running as 'hbase' and the DN is running as 'hdfs'. Then they would share a common group and have block files chmodded g+r. My understanding is that, both hbase region server and DN are running as the same user and not done using group access. Sanjay any comments? With the path cache, why's this true? It seems that, so long as you're accessing a relatively bounded number of blocks, the paths will all be cached and you won't need to re-RPC unless a block moves, etc? How do you assure relatively bounded number of blocks? Not sure what the HBase usecase here is. Additional information on the usage pattern would help. For every block to fill up the cache you end up creating an RPC proxy and a connection. The new patch is much cleaner. Also when I ran my client accessing 1000 blocks, I saw OutOfMemoryErrors.
        Hide
        Todd Lipcon added a comment -

        Since both the client and HDFS are running as the same user, this mechanism will work fine

        Could you explain that? I imagine the more likely deployment is that the client is running as 'hbase' and the DN is running as 'hdfs'. Then they would share a common group and have block files chmodded g+r.

        The ClientDatanodeProtocol RPC proxy is created using block token. This means an RPC proxy is created for every block. This robs the performance gain from the patch.

        With the path cache, why's this true? It seems that, so long as you're accessing a relatively bounded number of blocks, the paths will all be cached and you won't need to re-RPC unless a block moves, etc?

        Show
        Todd Lipcon added a comment - Since both the client and HDFS are running as the same user, this mechanism will work fine Could you explain that? I imagine the more likely deployment is that the client is running as 'hbase' and the DN is running as 'hdfs'. Then they would share a common group and have block files chmodded g+r. The ClientDatanodeProtocol RPC proxy is created using block token. This means an RPC proxy is created for every block. This robs the performance gain from the patch. With the path cache, why's this true? It seems that, so long as you're accessing a relatively bounded number of blocks, the paths will all be cached and you won't need to re-RPC unless a block moves, etc?
        Hide
        Suresh Srinivas added a comment -

        Updated patch.

        Show
        Suresh Srinivas added a comment - Updated patch.
        Hide
        Suresh Srinivas added a comment -

        The patch has few performance issues when ported to 0.20-security due to the security mechanism.

        1. The ClientDatanodeProtocol RPC proxy is created using block token. This means an RPC proxy is created for every block. This robs the performance gain from the patch.
        2. getBlockPathInfo() is authorized using block token. This makes the use of cache for block to path info confusing. A cache could be added an entry based on one token and an access to the cache using another token finds the cached path info.

        The new patch makes the following changes:

        1. A client needs a valid, long lived ClientDatanodeProtocol proxy, instead of creating one for each block access. In order to do that, kerberos auth method is used for RPC connectivity. Since both the client and HDFS are running as the same user, this mechanism will work fine.
        2. In Datanode#getBlockInfo() following checks are made:
          • The user corresponding to the caller of this method must match the user the datanode is running as.
          • When security enabled, the server allows only kerberos as the auth method.
        3. BlockReaderLocal allows short circuit read only for the user corresponding to the datanode.

        Pending:
        Will run more tests in secure setup and also will do some more benchmarking in both secure and non-secuire setup.

        Show
        Suresh Srinivas added a comment - The patch has few performance issues when ported to 0.20-security due to the security mechanism. The ClientDatanodeProtocol RPC proxy is created using block token. This means an RPC proxy is created for every block. This robs the performance gain from the patch. getBlockPathInfo() is authorized using block token. This makes the use of cache for block to path info confusing. A cache could be added an entry based on one token and an access to the cache using another token finds the cached path info. The new patch makes the following changes: A client needs a valid, long lived ClientDatanodeProtocol proxy, instead of creating one for each block access. In order to do that, kerberos auth method is used for RPC connectivity. Since both the client and HDFS are running as the same user, this mechanism will work fine. In Datanode#getBlockInfo() following checks are made: The user corresponding to the caller of this method must match the user the datanode is running as. When security enabled, the server allows only kerberos as the auth method. BlockReaderLocal allows short circuit read only for the user corresponding to the datanode. Pending: Will run more tests in secure setup and also will do some more benchmarking in both secure and non-secuire setup.
        Hide
        Suresh Srinivas added a comment -

        Now BlockMetadaDataHeader becomes a public class. What's the "real" visibility of that class?

        So it can be used outside the package, in DFSClient.

        I don;t understand it very well, but why is generation stamp ignored when reading from local ?

        Not sure which code you are talking about here

        What is the correct behavior if metadata versions do not match ? (currently you ignore checksums, but still read the data.)

        It is just cut and paste from BlockSender. I think metadata version has not changed in a while. At both these places, we should make changes with clearly defined behavior in a separate issue.

        Minor nit: soft lru cache is not specific to hdfs, and should be moved to common.
        Sounds good. I was doing that in my new patch.

        Show
        Suresh Srinivas added a comment - Now BlockMetadaDataHeader becomes a public class. What's the "real" visibility of that class? So it can be used outside the package, in DFSClient. I don;t understand it very well, but why is generation stamp ignored when reading from local ? Not sure which code you are talking about here What is the correct behavior if metadata versions do not match ? (currently you ignore checksums, but still read the data.) It is just cut and paste from BlockSender. I think metadata version has not changed in a while. At both these places, we should make changes with clearly defined behavior in a separate issue. Minor nit: soft lru cache is not specific to hdfs, and should be moved to common. Sounds good. I was doing that in my new patch.
        Hide
        Milind Bhandarkar added a comment -

        @suresh, sorry I wasted my time looking at an older version.

        Now BlockMetadaDataHeader becomes a public class. What's the "real" visibility of that class ?

        ClientdatanodeProtocol version is not changed.

        I don;t understand it very well, but why is generation stamp ignored when reading from local ?

        What is the correct behavior if metadata versions do not match ? (currently you ignore checksums, but still read the data.)

        Minor nit: soft lru cache is not specific to hdfs, and should be moved to common.

        Show
        Milind Bhandarkar added a comment - @suresh, sorry I wasted my time looking at an older version. Now BlockMetadaDataHeader becomes a public class. What's the "real" visibility of that class ? ClientdatanodeProtocol version is not changed. I don;t understand it very well, but why is generation stamp ignored when reading from local ? What is the correct behavior if metadata versions do not match ? (currently you ignore checksums, but still read the data.) Minor nit: soft lru cache is not specific to hdfs, and should be moved to common.
        Hide
        Suresh Srinivas added a comment -

        Milind, have you looked at the latest patch?

        Show
        Suresh Srinivas added a comment - Milind, have you looked at the latest patch?
        Hide
        Milind Bhandarkar added a comment -

        @Eli, I think Konst kept HDFS-2285 open for trunk commit.

        I took a brief look at this patch, and relying on a substring in the exception message is the anti-pattern used here. Can a subclass of InvalidBlockTokenException be created that specifically indicates token expired, and ex checked as instanceof of that ?

        Show
        Milind Bhandarkar added a comment - @Eli, I think Konst kept HDFS-2285 open for trunk commit. I took a brief look at this patch, and relying on a substring in the exception message is the anti-pattern used here. Can a subclass of InvalidBlockTokenException be created that specifically indicates token expired, and ex checked as instanceof of that ?
        Hide
        Eli Collins added a comment -

        @Arun, no but I think we should respect the 20x guidelines that Owen and Matt established, in particular that features go into trunk prior to branch date.

        Wrt HDFS-2285 if 22 contains a lot of stuff like this I wouldn't vote +1 for the release.

        Show
        Eli Collins added a comment - @Arun, no but I think we should respect the 20x guidelines that Owen and Matt established, in particular that features go into trunk prior to branch date. Wrt HDFS-2285 if 22 contains a lot of stuff like this I wouldn't vote +1 for the release.
        Hide
        Todd Lipcon added a comment -

        Yes, my opinion is that all patches should go into trunk first. The exceptions would be:

        • if the patch makes no sense on trunk, since it's superceded by another implementation or already fixed by some other JIRA (eg the 0.20-append stuff is superceded by the new impl on trunk, and many MR patches no longer make sense with MR2)
        • if the contributor commits to getting it into trunk within a very close timeline. For example, I'd be OK with this going in 0.20.205 if the patch authors commit to getting a finished patch available for trunk by the end of the following week. If this doesn't end up happening, then I'll be less likely to be OK with such an exception next time
        Show
        Todd Lipcon added a comment - Yes, my opinion is that all patches should go into trunk first. The exceptions would be: if the patch makes no sense on trunk, since it's superceded by another implementation or already fixed by some other JIRA (eg the 0.20-append stuff is superceded by the new impl on trunk, and many MR patches no longer make sense with MR2) if the contributor commits to getting it into trunk within a very close timeline. For example, I'd be OK with this going in 0.20.205 if the patch authors commit to getting a finished patch available for trunk by the end of the following week. If this doesn't end up happening, then I'll be less likely to be OK with such an exception next time
        Hide
        Arun C Murthy added a comment -

        Eli/Todd - Are you planning on -1'ing any patch which goes into a branch without going into trunk first? What about HDFS-2285? (Possibly HDFS-2271? not clear to me)

        Show
        Arun C Murthy added a comment - Eli/Todd - Are you planning on -1'ing any patch which goes into a branch without going into trunk first? What about HDFS-2285 ? (Possibly HDFS-2271 ? not clear to me)
        Hide
        Suresh Srinivas added a comment -

        Nathan, I added some code to TestShortCircuitLocalRead to enable bench marks. Please see the the documentation in the class on how to run it. These are the results of benchmarks I observed:

        Threads 1: Shortcircuit 22010, Regular 25292
        Threads 2: Shortcircuit 22110, Regular 28455
        Threads 3: Shortcircuit 24406, Regular 30012
        Threads 4: Shortcircuit 30469, Regular 48898
        Threads 10: Shortcircuit 73692, Regular 109824

        Beyond 3 threads, in regular reads I see bunch of IOExceptions (bind failed, could not find datanode with the block etc.) that I have not dug into.

        Show
        Suresh Srinivas added a comment - Nathan, I added some code to TestShortCircuitLocalRead to enable bench marks. Please see the the documentation in the class on how to run it. These are the results of benchmarks I observed: Threads 1: Shortcircuit 22010, Regular 25292 Threads 2: Shortcircuit 22110, Regular 28455 Threads 3: Shortcircuit 24406, Regular 30012 Threads 4: Shortcircuit 30469, Regular 48898 Threads 10: Shortcircuit 73692, Regular 109824 Beyond 3 threads, in regular reads I see bunch of IOExceptions (bind failed, could not find datanode with the block etc.) that I have not dug into.
        Hide
        Suresh Srinivas added a comment -

        Here is an updated patch with the following changes:

        1. Addressed all Todd's comments
        2. Made variables final where necessary
        3. Changed "dfs.read.shortcircuit." to "dfs.client.read.shortcircuit" and moved it to DFSConfigKeys.
        4. TestFileLocalRead.java
          • Renamed the test to TestShortCircuitLocalRead.java
          • Added class javadoc that describes the test correctly.
          • Added call to checkFullFile() to trigger local reads and to verify local read works.
          • Removed unused method checkFile
        5. DFSClient.java
          • Changed the local host check to more comprehensive check, which will handle multiple interfaces better. Please review this part closely.
          • Made changes to unwrap InvalidToken from RemoteException when making datanode#getBlockPathInfo() call.
        6. ClientDataNodeProtocol.java - reverted changing the versionID for newly added method.
        Show
        Suresh Srinivas added a comment - Here is an updated patch with the following changes: Addressed all Todd's comments Made variables final where necessary Changed "dfs.read.shortcircuit." to "dfs.client.read.shortcircuit" and moved it to DFSConfigKeys. TestFileLocalRead.java Renamed the test to TestShortCircuitLocalRead.java Added class javadoc that describes the test correctly. Added call to checkFullFile() to trigger local reads and to verify local read works. Removed unused method checkFile DFSClient.java Changed the local host check to more comprehensive check, which will handle multiple interfaces better. Please review this part closely. Made changes to unwrap InvalidToken from RemoteException when making datanode#getBlockPathInfo() call. ClientDataNodeProtocol.java - reverted changing the versionID for newly added method.
        Hide
        Nathan Roberts added a comment -

        Two comments:
        1) I really wish someone would provide some real benchmarks showing the benefit of this patch AFTER HDFS-941. Prior to HDFS-941, going through the datanode for everything was a nightmare and going direct to the file was a huge win. I'd really like to see a more recent comparison (apologies if someone did this comparison and I just missed it.)
        2) I'd like to see some comments somewhere about how this affects data visibility guarantees. Maybe around the definition of the new config option.

        Show
        Nathan Roberts added a comment - Two comments: 1) I really wish someone would provide some real benchmarks showing the benefit of this patch AFTER HDFS-941 . Prior to HDFS-941 , going through the datanode for everything was a nightmare and going direct to the file was a huge win. I'd really like to see a more recent comparison (apologies if someone did this comparison and I just missed it.) 2) I'd like to see some comments somewhere about how this affects data visibility guarantees. Maybe around the definition of the new config option.
        Hide
        Suresh Srinivas added a comment -

        I am reviewing the patch and making some changes. Will look look into your comment.

        Show
        Suresh Srinivas added a comment - I am reviewing the patch and making some changes. Will look look into your comment.
        Hide
        Todd Lipcon added a comment -

        Regarding the most recent patch:

        +              (targetAddr.getAddress().equals(localHost) ||
        +                  targetAddr.getHostName().startsWith("localhost"))) {
        

        Is it possible that getHostName here will incur a reverse DNS lookup and be problematic?

        +          // InvalidBlockTokenException is only thrown when the data pipeline
        +          // returns an error code. For local block reads, if the token has
        +          // expired we will get only a generic RemoteException from the RPC
        +          // to get the local path, and this requires the unfortunate hack
        +          // below.
        +          boolean isInvalidBlock = (ex instanceof InvalidBlockTokenException)
        +              || (ex.getMessage() != null && 
        +                    ex.getMessage().endsWith("is expired."));
        

        Can you refactor this repeated code and comment into a new function like boolean isExpiredBlockTokenException(Throwable t)?

        +          if (shortCircuitLocalReads && localHost != null &&
        +              (targetAddr.getAddress().equals(localHost) ||
        +                  targetAddr.getHostName().startsWith("localhost"))) {
        

        Refactor the above repeated code into a function like boolean shouldTryLocalRead(InetAddress addr)

        • The patch seems to change a number of logs from WARN level to INFO level. That seems like a separate change.
        +  /** Retrives the filename of the blockfile and the metafile from the datanode
        

        Typo "retrieves"

        
        +    if (isBlockTokenEnabled && UserGroupInformation.isSecurityEnabled()) {
        +      Set<TokenIdentifier> tokenIds = UserGroupInformation.getCurrentUser()
        +          .getTokenIdentifiers();
        +      if (tokenIds.size() != 1) {
        +        throw new IOException("Can't continue with getBlockPathInfo() "
        +            + "authorization since " + tokenIds.size() + " BlockTokenIdentifier "
        +            + "is found.");
        +      }
        +      for (TokenIdentifier tokenId : tokenIds) {
        +        BlockTokenIdentifier id = (BlockTokenIdentifier) tokenId;
        +        if (LOG.isDebugEnabled()) {
        +          LOG.debug("Got: " + id.toString());
        +        }
        +        blockTokenSecretManager.checkAccess(id, null, block,
        +            BlockTokenSecretManager.AccessMode.READ);
        +      }
        +    }
        

        This is all repeated code from recoverBlock - please refactor into a checkRpcAuthorizedForBlock() function

        +/**
        + * This class tests that a file need not be closed before its
        + * data can be read by another client.
        + */
        +public class TestFileLocalRead extends junit.framework.TestCase {
        

        This comment was copy-pasted from a different test case. Also, should use JUnit 4 for new tests.

        Show
        Todd Lipcon added a comment - Regarding the most recent patch: + (targetAddr.getAddress().equals(localHost) || + targetAddr.getHostName().startsWith( "localhost" ))) { Is it possible that getHostName here will incur a reverse DNS lookup and be problematic? + // InvalidBlockTokenException is only thrown when the data pipeline + // returns an error code. For local block reads, if the token has + // expired we will get only a generic RemoteException from the RPC + // to get the local path, and this requires the unfortunate hack + // below. + boolean isInvalidBlock = (ex instanceof InvalidBlockTokenException) + || (ex.getMessage() != null && + ex.getMessage().endsWith( "is expired." )); Can you refactor this repeated code and comment into a new function like boolean isExpiredBlockTokenException(Throwable t) ? + if (shortCircuitLocalReads && localHost != null && + (targetAddr.getAddress().equals(localHost) || + targetAddr.getHostName().startsWith( "localhost" ))) { Refactor the above repeated code into a function like boolean shouldTryLocalRead(InetAddress addr) The patch seems to change a number of logs from WARN level to INFO level. That seems like a separate change. + /** Retrives the filename of the blockfile and the metafile from the datanode Typo "retrieves" + if (isBlockTokenEnabled && UserGroupInformation.isSecurityEnabled()) { + Set<TokenIdentifier> tokenIds = UserGroupInformation.getCurrentUser() + .getTokenIdentifiers(); + if (tokenIds.size() != 1) { + throw new IOException( "Can't continue with getBlockPathInfo() " + + "authorization since " + tokenIds.size() + " BlockTokenIdentifier " + + "is found." ); + } + for (TokenIdentifier tokenId : tokenIds) { + BlockTokenIdentifier id = (BlockTokenIdentifier) tokenId; + if (LOG.isDebugEnabled()) { + LOG.debug( "Got: " + id.toString()); + } + blockTokenSecretManager.checkAccess(id, null , block, + BlockTokenSecretManager.AccessMode.READ); + } + } This is all repeated code from recoverBlock - please refactor into a checkRpcAuthorizedForBlock() function +/** + * This class tests that a file need not be closed before its + * data can be read by another client. + */ + public class TestFileLocalRead extends junit.framework.TestCase { This comment was copy-pasted from a different test case. Also, should use JUnit 4 for new tests.
        Hide
        Eli Collins added a comment -

        Per 20x release process "All new features must be added to trunk prior to the branch date". Ie this definitely shouldn't go in branch-20-security before trunk.

        Show
        Eli Collins added a comment - Per 20x release process "All new features must be added to trunk prior to the branch date". Ie this definitely shouldn't go in branch-20-security before trunk.
        Hide
        Todd Lipcon added a comment -

        Sure, I think we could commit this to trunk as an interim solution. Then we could also commit it to 0.20. I'll still be -0 since I think there is an implementation that doesn't have the security issues, but I won't veto. Something's better than nothing, like you said.

        Show
        Todd Lipcon added a comment - Sure, I think we could commit this to trunk as an interim solution. Then we could also commit it to 0.20. I'll still be -0 since I think there is an implementation that doesn't have the security issues, but I won't veto. Something's better than nothing, like you said.
        Hide
        dhruba borthakur added a comment -

        I see, is there a reason why we should not commit this to trunk (and 0.23 if needed)? (This will help those Hbase sites that either do not need security or where the HDFS&HBase trust domains are the same.)

        Show
        dhruba borthakur added a comment - I see, is there a reason why we should not commit this to trunk (and 0.23 if needed)? (This will help those Hbase sites that either do not need security or where the HDFS&HBase trust domains are the same.)
        Hide
        Todd Lipcon added a comment -

        It's not strictly technical, but I am -1 on principle on any patch that adds a major feature to 0.20.x that isn't available in trunk. I agree that the short-circuit idea is a good one (I was one of the original proponents of the idea 2 years ago if you recall from HDFS-347).

        Regarding this particular patch, I think it is technically troublesome since it can only function by opening the trust domain of HDFS to include HBase (or any other client that wants to use the feature). There are block tokens, but they're essentially "advisory" as you can easily scan the HDFS data directories once you have read access on the local FS. I'm willing to overlook this so long as people are willing to review and eventually move to the actually secure solution I proposed long ago (and implemented though long since stale!) in HDFS-347. But even if this is an interim solution, we need a (interim or not) solution for 0.23 as well.

        Show
        Todd Lipcon added a comment - It's not strictly technical, but I am -1 on principle on any patch that adds a major feature to 0.20.x that isn't available in trunk. I agree that the short-circuit idea is a good one (I was one of the original proponents of the idea 2 years ago if you recall from HDFS-347 ). Regarding this particular patch, I think it is technically troublesome since it can only function by opening the trust domain of HDFS to include HBase (or any other client that wants to use the feature). There are block tokens, but they're essentially "advisory" as you can easily scan the HDFS data directories once you have read access on the local FS. I'm willing to overlook this so long as people are willing to review and eventually move to the actually secure solution I proposed long ago (and implemented though long since stale!) in HDFS-347 . But even if this is an interim solution, we need a (interim or not) solution for 0.23 as well.
        Hide
        dhruba borthakur added a comment -

        Todd: this patch seems to be the best way to move the support for HBase into an released version of HDFS. It improves the performance of HDFS tremendously. This patch also does not affect the acceptance/rejection of any other patches you are referring to.

        Are you -1 this patch on any technical grounds?

        Show
        dhruba borthakur added a comment - Todd: this patch seems to be the best way to move the support for HBase into an released version of HDFS. It improves the performance of HDFS tremendously. This patch also does not affect the acceptance/rejection of any other patches you are referring to. Are you -1 this patch on any technical grounds?
        Hide
        Todd Lipcon added a comment -

        Is this also going to be developed for trunk? I am -1 for committing this to a branch until it is also committed to trunk. I am -0 on committing this version of the patch to trunk when there are patches that have been pending review for months working towards the unix-socket based method that avoids the hack of allowing the hbase user to access all block files directly.

        Show
        Todd Lipcon added a comment - Is this also going to be developed for trunk? I am -1 for committing this to a branch until it is also committed to trunk. I am -0 on committing this version of the patch to trunk when there are patches that have been pending review for months working towards the unix-socket based method that avoids the hack of allowing the hbase user to access all block files directly.
        Hide
        Sanjay Radia added a comment -
        • Patch updated to 20-security.
        • Added block token check
          • ignore my comment about block token parameter - it is passed as part of connection. Added code to check the token.
        Show
        Sanjay Radia added a comment - Patch updated to 20-security. Added block token check ignore my comment about block token parameter - it is passed as part of connection. Added code to check the token.
        Hide
        Sanjay Radia added a comment -

        getBlockPathInfo(block) - add block access token parameter which is verified on the data node.

        Show
        Sanjay Radia added a comment - getBlockPathInfo(block) - add block access token parameter which is verified on the data node.
        Hide
        Andrew Purtell added a comment -

        A revised commit without accidentally included .orig files from patch: https://github.com/trendmicro/hadoop-common/commit/c19169f9c93f89ddafd6b6293a3e327a9741dfb0

        Show
        Andrew Purtell added a comment - A revised commit without accidentally included .orig files from patch: https://github.com/trendmicro/hadoop-common/commit/c19169f9c93f89ddafd6b6293a3e327a9741dfb0
        Hide
        Andrew Purtell added a comment -

        Attached is an 0.20-ish patch ported from our production tree to an ASF derivative: https://github.com/trendmicro/hadoop-common/tree/0.20-security-append-HDFS-347

        (The 0.20-security-append tree, based on security-204, is here: https://github.com/trendmicro/hadoop-common/tree/0.20-security-append)

        The attached patch is this change: https://github.com/trendmicro/hadoop-common/commit/18971523a9260acbe920c3da0c5a2623eacec1d2

        This was initially Ryan Rawson's patch for 0.20-append on HDFS-347, later merged with the checksumming improvements from Facebook, and merged with our enhancements and fixups for security and metrics v2. It includes a new metric for the number of times clients successful get a local path to block data.

        Show
        Andrew Purtell added a comment - Attached is an 0.20-ish patch ported from our production tree to an ASF derivative: https://github.com/trendmicro/hadoop-common/tree/0.20-security-append-HDFS-347 (The 0.20-security-append tree, based on security-204, is here: https://github.com/trendmicro/hadoop-common/tree/0.20-security-append ) The attached patch is this change: https://github.com/trendmicro/hadoop-common/commit/18971523a9260acbe920c3da0c5a2623eacec1d2 This was initially Ryan Rawson's patch for 0.20-append on HDFS-347 , later merged with the checksumming improvements from Facebook, and merged with our enhancements and fixups for security and metrics v2. It includes a new metric for the number of times clients successful get a local path to block data.
        Hide
        Sanjay Radia added a comment -

        HDFS-347 discusses ways to optimize reads for local clients. A clean design is fairly involved. A shortcut has been proposed
        where the client access the hdfs file blocks directly; this works if the client is the same user/group as the DN daemon.
        This is non-invasive and is a good short term solution till HDFS-347 is completed.

        Show
        Sanjay Radia added a comment - HDFS-347 discusses ways to optimize reads for local clients. A clean design is fairly involved. A shortcut has been proposed where the client access the hdfs file blocks directly; this works if the client is the same user/group as the DN daemon. This is non-invasive and is a good short term solution till HDFS-347 is completed.

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Sanjay Radia
          • Votes:
            0 Vote for this issue
            Watchers:
            45 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development