Hadoop Common
  1. Hadoop Common
  2. HADOOP-2323

JobTracker.close() prints stack traces for exceptions that are not errors

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      JobTracker.close() prints a stack trace for an interrupted exception even though it was the method that interrupted the thread that threw the exception. For example:

            this.expireTrackers.stopTracker();
            try {
              this.expireTrackersThread.interrupt();
              this.expireTrackersThread.join();
            } catch (InterruptedException ex) {
              ex.printStackTrace();
            }
      

      Well of course it is going to catch an InterruptedException after it just interrupted the thread!

      This is not an error and should not be dumped to the logs!

      In other circumstances, catching InterruptedException is entirely appropriate. Just not in close where you've told the thread to shutdown and then interrupted it to ensure it does!

      1. patch.txt
        4 kB
        Jim Kellerman
      2. patch.txt
        6 kB
        Jim Kellerman
      3. patch.txt
        6 kB
        Jim Kellerman
      4. patch.txt
        7 kB
        Jim Kellerman
      5. patch.txt
        7 kB
        Jim Kellerman

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-Nightly #333 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/333/ )
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12371708/patch.txt
        against trunk revision r604340.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12371708/patch.txt against trunk revision r604340. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1352/console This message is automatically generated.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Jim!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Jim!
        Hide
        Jim Kellerman added a comment -

        For Hudson's approval.

        Show
        Jim Kellerman added a comment - For Hudson's approval.
        Hide
        Jim Kellerman added a comment -

        This time without absolute path names.

        Show
        Jim Kellerman added a comment - This time without absolute path names.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12371706/patch.txt
        against trunk revision r604325.

        @author +1. The patch does not contain any @author tags.

        patch -1. The patch command could not apply the patch.

        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1351/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12371706/patch.txt against trunk revision r604325. @author +1. The patch does not contain any @author tags. patch -1. The patch command could not apply the patch. Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1351/console This message is automatically generated.
        Hide
        Jim Kellerman added a comment -

        Patch had fully qualified path names in it. Duh!

        Show
        Jim Kellerman added a comment - Patch had fully qualified path names in it. Duh!
        Hide
        Jim Kellerman added a comment -

        Submit for Hudson's approval.

        Show
        Jim Kellerman added a comment - Submit for Hudson's approval.
        Hide
        Jim Kellerman added a comment -

        Revised patch that will apply on trunk.

        Show
        Jim Kellerman added a comment - Revised patch that will apply on trunk.
        Hide
        Jim Kellerman added a comment -

        Patch no longer applies on trunk.

        Show
        Jim Kellerman added a comment - Patch no longer applies on trunk.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12370806/patch.txt
        against trunk revision r600443.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12370806/patch.txt against trunk revision r600443. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1243/console This message is automatically generated.
        Hide
        Jim Kellerman added a comment -

        It is likely that the failure of one of the two tests TestScanner2, was caused by HADOOP-2331.
        Will try test run again after that is fixed.

        Show
        Jim Kellerman added a comment - It is likely that the failure of one of the two tests TestScanner2, was caused by HADOOP-2331 . Will try test run again after that is fixed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12370806/patch.txt
        against trunk revision r600244.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12370806/patch.txt against trunk revision r600244. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1239/console This message is automatically generated.
        Hide
        Jim Kellerman added a comment -

        Fixed problems with previous patch, resubmitting.

        Show
        Jim Kellerman added a comment - Fixed problems with previous patch, resubmitting.
        Hide
        Jim Kellerman added a comment -

        Patch did not suppress messages as expected. Cancelling.

        Show
        Jim Kellerman added a comment - Patch did not suppress messages as expected. Cancelling.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12370805/patch.txt
        against trunk revision r600244.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12370805/patch.txt against trunk revision r600244. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1238/console This message is automatically generated.
        Hide
        Jim Kellerman added a comment -

        More minimal patch in which all threads exit in the same manner.

        Show
        Jim Kellerman added a comment - More minimal patch in which all threads exit in the same manner.
        Hide
        Jim Kellerman added a comment -

        Owen,

        I agree with you, but not knowing the internals of the JobTracker that well, did not know if there was a reason that JobTracker$ExpireTrackers and JobTracker$TaskCommitQueue were different from the other threads and intentionally let the exception escape rather than shut down.

        Most of the other threads also use a class-level boolean but that is probably not necessary. A boolean in the method (or just breaking out of the loop) would work just as well.

        I'll work up a more minimal patch along these lines and you (or someone else who knows JobTracker) can decide which patch you like.

        -Jim

        Show
        Jim Kellerman added a comment - Owen, I agree with you, but not knowing the internals of the JobTracker that well, did not know if there was a reason that JobTracker$ExpireTrackers and JobTracker$TaskCommitQueue were different from the other threads and intentionally let the exception escape rather than shut down. Most of the other threads also use a class-level boolean but that is probably not necessary. A boolean in the method (or just breaking out of the loop) would work just as well. I'll work up a more minimal patch along these lines and you (or someone else who knows JobTracker) can decide which patch you like. -Jim
        Hide
        Owen O'Malley added a comment -

        You've made a lot more changes than the jira description suggests.

        In general, I'd rather have threads interrupt other threads rather than set a volatile boolean flag, but I haven't looked at your patch in the real context, just standalone.

        Show
        Owen O'Malley added a comment - You've made a lot more changes than the jira description suggests. In general, I'd rather have threads interrupt other threads rather than set a volatile boolean flag, but I haven't looked at your patch in the real context, just standalone.
        Hide
        Jim Kellerman added a comment -

        Without this patch, the following messages appear in test output when there is no error and the JobTracker is just shutting down normally:

            [junit] 2007-12-01 12:04:03,105 ERROR [expireTrackers] org.apache.hadoop.mapred.JobTracker$ExpireTrackers.run(JobTracker.java:308): Tracker Expiry Thread got exception: java.lang.InterruptedException: sleep interrupted
            [junit] 	at java.lang.Thread.sleep(Native Method)
            [junit] 	at org.apache.hadoop.mapred.JobTracker$ExpireTrackers.run(JobTracker.java:263)
            [junit] 	at java.lang.Thread.run(Thread.java:595)
        
            [junit] 2007-12-01 12:04:03,106 WARN  [Task Commit Thread] org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2017): Task Commit Thread exiting, got interrupted: java.lang.InterruptedException
            [junit] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1815)
            [junit] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1850)
            [junit] 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
            [junit] 	at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:1919)
        

        with this patch, these error messages do not appear

        Show
        Jim Kellerman added a comment - Without this patch, the following messages appear in test output when there is no error and the JobTracker is just shutting down normally: [junit] 2007-12-01 12:04:03,105 ERROR [expireTrackers] org.apache.hadoop.mapred.JobTracker$ExpireTrackers.run(JobTracker.java:308): Tracker Expiry Thread got exception: java.lang.InterruptedException: sleep interrupted [junit] at java.lang. Thread .sleep(Native Method) [junit] at org.apache.hadoop.mapred.JobTracker$ExpireTrackers.run(JobTracker.java:263) [junit] at java.lang. Thread .run( Thread .java:595) [junit] 2007-12-01 12:04:03,106 WARN [Task Commit Thread ] org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2017): Task Commit Thread exiting, got interrupted: java.lang.InterruptedException [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1815) [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1850) [junit] at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359) [junit] at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:1919) with this patch, these error messages do not appear
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12370714/patch.txt
        against trunk revision r599951.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12370714/patch.txt against trunk revision r599951. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1223/console This message is automatically generated.

          People

          • Assignee:
            Jim Kellerman
            Reporter:
            Jim Kellerman
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development