Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2655

BlockReaderLocal#skip performs unnecessary IO

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.1
    • Fix Version/s: 0.23.2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Per HDFS-2654 BlockReaderLocal#skip performs the skip by reading the data so we stay in sync with checksums. This could be implemented more efficiently in the future to skip to the beginning of the appropriate checksum chunk and then only read to the middle of that chunk.

      1. HDFS-2655.txt
        5 kB
        Brandon Li
      2. HDFS-2655.txt
        5 kB
        Brandon Li

        Activity

        Hide
        Brandon Li added a comment -

        Modified BlockReaderLocal.skip() method to be more efficient:

        1. if the new position offset is already read, just reposition the dataBuff
        2. if the new position offset is just a small gap from the last read byte, only read the small gap
        3. if there is a big gap(>bytesPerChecksum) from last read byte to new position, discard the buffers, skip to the last he beginning of the appropriate checksum chunk and then read to the middle of that chunk to be in sync with checksums.

        A test is also added to test skip function.

        Show
        Brandon Li added a comment - Modified BlockReaderLocal.skip() method to be more efficient: 1. if the new position offset is already read, just reposition the dataBuff 2. if the new position offset is just a small gap from the last read byte, only read the small gap 3. if there is a big gap(>bytesPerChecksum) from last read byte to new position, discard the buffers, skip to the last he beginning of the appropriate checksum chunk and then read to the middle of that chunk to be in sync with checksums. A test is also added to test skip function.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12511209/HDFS-2655.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1868//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511209/HDFS-2655.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1868//console This message is automatically generated.
        Hide
        Brandon Li added a comment -

        rebased the patch on current trunk

        Show
        Brandon Li added a comment - rebased the patch on current trunk
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12514747/HDFS-2655.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1873//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1873//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514747/HDFS-2655.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1873//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1873//console This message is automatically generated.
        Hide
        Jitendra Nath Pandey added a comment -

        +1

        Show
        Jitendra Nath Pandey added a comment - +1
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1811 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1811/)
        HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1811 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1811/ ) HDFS-2655 . BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1737/)
        HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1737 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1737/ ) HDFS-2655 . BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Jitendra Nath Pandey added a comment -

        This is committed. Thanks to Brandon!

        Show
        Jitendra Nath Pandey added a comment - This is committed. Thanks to Brandon!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #548 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/548/)
        Merged 1245118 from trunk for HDFS-2655. (Revision 1245122)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #548 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/548/ ) Merged 1245118 from trunk for HDFS-2655 . (Revision 1245122) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #561 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/561/)
        Merged 1245118 from trunk for HDFS-2655. (Revision 1245122)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #561 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/561/ ) Merged 1245118 from trunk for HDFS-2655 . (Revision 1245122) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1749 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1749/)
        HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118)

        Result = ABORTED
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1749 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1749/ ) HDFS-2655 . BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118) Result = ABORTED jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #564 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/564/)
        Merged 1245118 from trunk for HDFS-2655. (Revision 1245122)

        Result = ABORTED
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #564 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/564/ ) Merged 1245118 from trunk for HDFS-2655 . (Revision 1245122) Result = ABORTED jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #171 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/171/)
        Merged 1245118 from trunk for HDFS-2655. (Revision 1245122)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #171 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/171/ ) Merged 1245118 from trunk for HDFS-2655 . (Revision 1245122) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/958/)
        HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #958 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/958/ ) HDFS-2655 . BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/199/)
        Merged 1245118 from trunk for HDFS-2655. (Revision 1245122)

        Result = FAILURE
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #199 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/199/ ) Merged 1245118 from trunk for HDFS-2655 . (Revision 1245122) Result = FAILURE jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245122 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #993 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/993/)
        HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118)

        Result = SUCCESS
        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #993 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/993/ ) HDFS-2655 . BlockReaderLocal#skip performs unnecessary IO. Contributed by Brandon Li. (Revision 1245118) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1245118 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java

          People

          • Assignee:
            Brandon Li
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development