Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-532

Allow applications to know that a read request failed because block is missing

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I have an application that has intelligence to retrieve data from alternate locations if HDFS cannot provide this data. This can happen when data in HDFS is corrupted or the block is missing. HDFS already throws ChecksumException if the block is corrupted and throws a generic IOException if the block is missing. I would like HDFS to throw BlockMissingException when a read request encounters a block that has no locations associated with it.

      1. BlockMissingException.patch
        9 kB
        dhruba borthakur
      2. BlockMissingException2.txt
        9 kB
        dhruba borthakur

        Activity

        dhruba borthakur created issue -
        dhruba borthakur made changes -
        Field Original Value New Value
        Fix Version/s 0.21.0 [ 12314046 ]
        Component/s hdfs client [ 12312928 ]
        dhruba borthakur made changes -
        Summary Allow applications to know that a read failed beucase block is missing Allow applications to know that a read request failed because block is missing
        Hide
        dhruba borthakur added a comment -

        if a read request encounters a missing block, then raise a BlockMissingException. BlockMissingException is derived from IOException.

        Show
        dhruba borthakur added a comment - if a read request encounters a missing block, then raise a BlockMissingException. BlockMissingException is derived from IOException.
        dhruba borthakur made changes -
        Attachment BlockMissingException.patch [ 12415799 ]
        Hide
        Zheng Shao added a comment -

        +1

        Show
        Zheng Shao added a comment - +1
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12415799/BlockMissingException.patch
        against trunk revision 803622.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 96 javac compiler warnings (more than the trunk's current 95 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415799/BlockMissingException.patch against trunk revision 803622. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 96 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        Need to fix the javac warning introduced by this patch.

        Show
        dhruba borthakur added a comment - Need to fix the javac warning introduced by this patch.
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        dhruba borthakur added a comment -

        Fix javac warning in earlier patch.

        Show
        dhruba borthakur added a comment - Fix javac warning in earlier patch.
        dhruba borthakur made changes -
        Attachment BlockMissingException2.txt [ 12416405 ]
        Hide
        dhruba borthakur added a comment -

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        dhruba borthakur added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12416405/BlockMissingException2.txt
        against trunk revision 803622.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416405/BlockMissingException2.txt against trunk revision 803622. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/65/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        I just committed this.

        Show
        dhruba borthakur added a comment - I just committed this.
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #49 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/49/)
        . Allow applications to know that a read request failed
        because block is missing. (dhruba)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #49 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/49/ ) . Allow applications to know that a read request failed because block is missing. (dhruba)
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        14h 40m 1 dhruba borthakur 13/Aug/09 07:29
        Open Open Patch Available Patch Available
        5d 20h 24m 2 dhruba borthakur 13/Aug/09 07:37
        Patch Available Patch Available Resolved Resolved
        11h 33m 1 dhruba borthakur 13/Aug/09 19:11
        Resolved Resolved Closed Closed
        376d 2h 37m 1 Tom White 24/Aug/10 21:48

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development