Hadoop Common
  1. Hadoop Common
  2. HADOOP-7140

IPC Reader threads do not stop when server stops

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 1-win, 1.3.0
    • Fix Version/s: 0.22.0, 1.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After HADOOP-6713, the new IPC "Reader" threads are not properly stopped when the server shuts down. One repercussion of this is that conditions that are supposed to shut down a daemon no longer work (eg the TT doesn't shut itself down if it detects an incompatible build version)

      1. HADOOP-7140.branch-1-win.patch
        5 kB
        Ivan Mitic
      2. HADOOP-7140.branch-1.patch
        5 kB
        Ivan Mitic
      3. hadoop-7140.txt
        5 kB
        Todd Lipcon
      4. hadoop-7140.txt
        5 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Ivan Mitic added a comment -

          Patch committed to branch-1.

          Show
          Ivan Mitic added a comment - Patch committed to branch-1.
          Hide
          Ivan Mitic added a comment -

          Thanks Chris! Will commit to branch-1 shortly.

          Since we'll be merging branch-1 changes into branch-1-win, I'll wait for the merge to pick up this change.

          Show
          Ivan Mitic added a comment - Thanks Chris! Will commit to branch-1 shortly. Since we'll be merging branch-1 changes into branch-1-win, I'll wait for the merge to pick up this change.
          Hide
          Chris Nauroth added a comment -

          +1 for the backport patches. I ran TestRPC successfully with the backport.

          Show
          Chris Nauroth added a comment - +1 for the backport patches. I ran TestRPC successfully with the backport.
          Hide
          Ivan Mitic added a comment -

          Attaching the branch-1 and branch-1-win compatible patches.

          Show
          Ivan Mitic added a comment - Attaching the branch-1 and branch-1-win compatible patches.
          Hide
          Ivan Mitic added a comment -

          Looks like we merged HADOOP-6713 to branch-1 but missed this Jira. I run into an issue where TT didn't shut itself down because of running IPC reader threads.

          Reopening for backport to branch-1 and branch-1-win.

          Show
          Ivan Mitic added a comment - Looks like we merged HADOOP-6713 to branch-1 but missed this Jira. I run into an issue where TT didn't shut itself down because of running IPC reader threads. Reopening for backport to branch-1 and branch-1-win.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #607 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/607/)
          HADOOP-7140. IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #607 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/607/ ) HADOOP-7140 . IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #25 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/25/)
          HADOOP-7140. IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #25 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/25/ ) HADOOP-7140 . IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #503 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/503/)
          HADOOP-7140. IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #503 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/503/ ) HADOOP-7140 . IPC Reader threads do not stop when server stops. Contributed by Todd Lipcon.
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and 22, thanks for reviewing, Tom.

          Show
          Todd Lipcon added a comment - Committed to trunk and 22, thanks for reviewing, Tom.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471020/hadoop-7140.txt
          against trunk revision 1070021.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471020/hadoop-7140.txt against trunk revision 1070021. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/235//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          New rev of the patch:

          • improves test assertions as suggested by Tom
          • adds a try{}finally{} in the test so as to avoid causing problems if the test fails

          Tom and I discussed his idea about interrupts offline - we decided that using interrupt() is a bit harsh since we don't want to catch a reader in the middle of a readAndProcess(). Using selector.wakeup is cleaner.

          Show
          Todd Lipcon added a comment - New rev of the patch: improves test assertions as suggested by Tom adds a try{}finally{} in the test so as to avoid causing problems if the test fails Tom and I discussed his idea about interrupts offline - we decided that using interrupt() is a bit harsh since we don't want to catch a reader in the middle of a readAndProcess(). Using selector.wakeup is cleaner.
          Hide
          Tom White added a comment -

          This looks fine, but I wonder if it would be simpler to act on the InterruptedException in Reader (rather than just log it as the code currently does) so the thread exits when the pool is shutdown?

          Minor nit: add messages to the test assertions so that the case of zero running threads can be more easily distinguished for the before and after case.

          Show
          Tom White added a comment - This looks fine, but I wonder if it would be simpler to act on the InterruptedException in Reader (rather than just log it as the code currently does) so the thread exits when the pool is shutdown? Minor nit: add messages to the test assertions so that the case of zero running threads can be more easily distinguished for the before and after case.
          Hide
          Todd Lipcon added a comment -

          This patch also fixes HADOOP-7132, since it was simple to fix while I was at it.

          Show
          Todd Lipcon added a comment - This patch also fixes HADOOP-7132 , since it was simple to fix while I was at it.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12470978/hadoop-7140.txt
          against trunk revision 1070021.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470978/hadoop-7140.txt against trunk revision 1070021. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/234//console This message is automatically generated.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development