Hadoop Common
  1. Hadoop Common
  2. HADOOP-9394

Port findHangingTest.sh from HBase to Hadoop

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HBase has this handy dev-support/findHangingTests.sh script, which parses Jenkins consoleText and finds hanging tests for you. This has been especially useful for identifying balancer test timeouts (see HDFS-4376 and HDFS-4261). It'd be nice to have this in our own dev-support directory.

        Issue Links

          Activity

          Hide
          Andrew Wang added a comment -

          Patch attached. It's just a cp from hbase trunk to here, I've used this script multiple times myself successfully.

          No tests for obvious reasons.

          Show
          Andrew Wang added a comment - Patch attached. It's just a cp from hbase trunk to here, I've used this script multiple times myself successfully. No tests for obvious reasons.
          Hide
          David S. Wang added a comment -

          +1 (non-binding)

          Show
          David S. Wang added a comment - +1 (non-binding)
          Hide
          Ted Yu added a comment -

          One enhancement for the script is that we should detect interleaving of tests in console output.
          e.g. from https://builds.apache.org/job/PreCommit-HBASE-Build/4760/console :

          Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
          Running org.apache.hadoop.hbase.regionserver.TestStore
          Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.224 sec
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 188.814 sec
          

          The two tests actually both passed. However, the script would think that TestChangingEncoding hung.

          Show
          Ted Yu added a comment - One enhancement for the script is that we should detect interleaving of tests in console output. e.g. from https://builds.apache.org/job/PreCommit-HBASE-Build/4760/console : Running org.apache.hadoop.hbase.io.encoding.TestChangingEncoding Running org.apache.hadoop.hbase.regionserver.TestStore Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.224 sec Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 188.814 sec The two tests actually both passed. However, the script would think that TestChangingEncoding hung.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573179/hadoop-9394-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2312//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2312//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573179/hadoop-9394-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2312//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2312//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          Hi Ted,

          Could we do that fix in a follow-on JIRA? The script is already quite useful in its current form.

          Feel free to report in HBase and Hadoop and link together.

          Show
          Andrew Wang added a comment - Hi Ted, Could we do that fix in a follow-on JIRA? The script is already quite useful in its current form. Feel free to report in HBase and Hadoop and link together.
          Hide
          Ted Yu added a comment -

          follow-on JIRA is fine for enhancement.

          Show
          Ted Yu added a comment - follow-on JIRA is fine for enhancement.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I'm going to commit this momentarily.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk.

          Thanks a lot for the contribution, Andrew.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk. Thanks a lot for the contribution, Andrew.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3450/)
          HADOOP-9394. Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419
          Files :

          • /hadoop/common/trunk/dev-support/findHangingTest.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3450 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3450/ ) HADOOP-9394 . Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419 Files : /hadoop/common/trunk/dev-support/findHangingTest.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #153 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/153/)
          HADOOP-9394. Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419
          Files :

          • /hadoop/common/trunk/dev-support/findHangingTest.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #153 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/153/ ) HADOOP-9394 . Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419 Files : /hadoop/common/trunk/dev-support/findHangingTest.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1342 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1342/)
          HADOOP-9394. Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419
          Files :

          • /hadoop/common/trunk/dev-support/findHangingTest.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1342 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1342/ ) HADOOP-9394 . Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419 Files : /hadoop/common/trunk/dev-support/findHangingTest.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1370 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1370/)
          HADOOP-9394. Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419
          Files :

          • /hadoop/common/trunk/dev-support/findHangingTest.sh
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1370 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1370/ ) HADOOP-9394 . Port findHangingTest.sh from HBase to Hadoop. Contributed by Andrew Wang. (Revision 1455419) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1455419 Files : /hadoop/common/trunk/dev-support/findHangingTest.sh /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              Andrew Wang
              Reporter:
              Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development