Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2375

Improve exception thrown by SpillRecord for bad spill index

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None

      Description

      In trying to debug some issues on a cluster that was seeing shuffle fetch failures, I found the IOException thrown by the SpillRecord constructor to be insufficient. An improved exception would have more details about the file path, offset, etc.

        Activity

        Todd Lipcon created issue -
        Hide
        Todd Lipcon added a comment -

        Here's a patch which includes some extra details in the exception message. I found this useful when narrowing down the cause of the fetch failures in my problematic cluster.

        Show
        Todd Lipcon added a comment - Here's a patch which includes some extra details in the exception message. I found this useful when narrowing down the cause of the fetch failures in my problematic cluster.
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment mapreduce-2375.txt [ 12473216 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Tom White added a comment -

        Ideally there wouldn't be an extra level of wrapping for ChecksumException (which is thrown then caught in the same method). Either add a new catch block and rethrow it, or log the extra detail in the IOException catch block and rethrow there.

        Show
        Tom White added a comment - Ideally there wouldn't be an extra level of wrapping for ChecksumException (which is thrown then caught in the same method). Either add a new catch block and rethrow it, or log the extra detail in the IOException catch block and rethrow there.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12473216/mapreduce-2375.txt
        against trunk revision 1079072.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473216/mapreduce-2375.txt against trunk revision 1079072. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/135//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Upon further investigation of this issue, it's looking more and more likely that it's a kernel bug, so going to close this one as wontfix.

        Show
        Todd Lipcon added a comment - Upon further investigation of this issue, it's looking more and more likely that it's a kernel bug, so going to close this one as wontfix.
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development