Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs-client, performance
    • Labels:
      None
    • Target Version/s:

      Description

      The read path through BlockReaderLocal does not take advantage of readahead or drop-behind in the way that BlockSender does. We could arguably stand to gain even more from hinting about read patterns to the kernel here, so we should add the same mechanisms to BlockReaderLocal.

        Issue Links

          Activity

          Henry Robinson created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Fix Version/s 3.0.0 [ 12320356 ]
          Target Version/s 3.0.0 [ 12320356 ]
          Component/s hdfs client [ 12312928 ]
          Component/s performance [ 12316501 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-2744 [ HDFS-2744 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to HDFS-2465 [ HDFS-2465 ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Mapreduce-trunk-Commit #2373 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2373/])
              Amendment to my previous commit, fixing JIRA in CHANGES.txt from HDFS-3228 to HDFS-3428 (Revision 1349556)

               Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349556
          Files :
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Hdfs-trunk-Commit #2423 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2423/])
              Amendment to my previous commit, fixing JIRA in CHANGES.txt from HDFS-3228 to HDFS-3428 (Revision 1349556)

               Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349556
          Files :
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Common-trunk-Commit #2350 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2350/])
              Amendment to my previous commit, fixing JIRA in CHANGES.txt from HDFS-3228 to HDFS-3428 (Revision 1349556)

               Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349556
          Files :
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Mapreduce-trunk-Commit #2372 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2372/])
              HDFS-3228. Move DelegationTokenRenewer to common (tucu) (Revision 1349555)

               Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349555
          Files :
          * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Common-trunk-Commit #2349 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2349/])
              HDFS-3228. Move DelegationTokenRenewer to common (tucu) (Revision 1349555)

               Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349555
          Files :
          * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Hdfs-trunk-Commit #2422 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2422/])
              HDFS-3228. Move DelegationTokenRenewer to common (tucu) (Revision 1349555)

               Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349555
          Files :
          * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          ]
          Todd Lipcon made changes -
          Comment [ These messages refer to HDFS-3428, not this issue. Looks like a typo in the commit msg. ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Mapreduce-trunk #1108 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1108/])
              Amendment to my previous commit, fixing JIRA in CHANGES.txt from HDFS-3228 to HDFS-3428 (Revision 1349556)
          HDFS-3228. Move DelegationTokenRenewer to common (tucu) (Revision 1349555)

               Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349556
          Files :
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349555
          Files :
          * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          ]
          Todd Lipcon made changes -
          Comment [ Integrated in Hadoop-Hdfs-trunk #1075 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1075/])
              Amendment to my previous commit, fixing JIRA in CHANGES.txt from HDFS-3228 to HDFS-3428 (Revision 1349556)
          HDFS-3228. Move DelegationTokenRenewer to common (tucu) (Revision 1349555)

               Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349556
          Files :
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349555
          Files :
          * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenRenewer.java
          * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          ]
          Todd Lipcon made changes -
          Link This issue is duplicated by HDFS-4697 [ HDFS-4697 ]
          Hide
          Colin Patrick McCabe added a comment -

          It's worth noting that BlockReaderLocal already implements a form of readahead, in that it reads and checksums dfs.client.read.shortcircuit.buffer.size bytes at a time. Typically this is 1 MB. (Since HDFS-5634, we will also read fewer bytes than dfs.client.read.shortcircuit.buffer.size if readahead is turned down or turned off.)

          Of course, this 1 MB read is done in the foreground (you don't get back any bytes until it finishes). An fadvise-based system would operate a little bit differently.

          Show
          Colin Patrick McCabe added a comment - It's worth noting that BlockReaderLocal already implements a form of readahead, in that it reads and checksums dfs.client.read.shortcircuit.buffer.size bytes at a time. Typically this is 1 MB. (Since HDFS-5634 , we will also read fewer bytes than dfs.client.read.shortcircuit.buffer.size if readahead is turned down or turned off.) Of course, this 1 MB read is done in the foreground (you don't get back any bytes until it finishes). An fadvise-based system would operate a little bit differently.

            People

            • Assignee:
              Henry Robinson
              Reporter:
              Henry Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development