Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11965

determine-flaky-tests needs a summary mode

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running determine-flaky-tests against PreCommit-HDFS-Build generates just slightly under 10,000 lines of output. That's significantly too much to be useful for the casual user. It's also not formatted in such a way to be easily machine parseable to generate a decent report. A summary mode should be added that just prints out the top X failed tests.

      1. HADOOP-11965.001.patch
        6 kB
        Yufei Gu
      2. HADOOP-11965.002.patch
        6 kB
        Yufei Gu
      3. HADOOP-11965.003.patch
        6 kB
        Yufei Gu

        Activity

        Hide
        yzhangal Yongjun Zhang added a comment -

        Thanks Allen Wittenauer for reporting the issue, I assigned it to myself and will try to work on it asap.

        Show
        yzhangal Yongjun Zhang added a comment - Thanks Allen Wittenauer for reporting the issue, I assigned it to myself and will try to work on it asap.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Yufei showed interest working on the improvement here, I'm assigning to him. Thanks Yufei.

        Show
        yzhangal Yongjun Zhang added a comment - Yufei showed interest working on the improvement here, I'm assigning to him. Thanks Yufei.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 17s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739719/HADOOP-11965.001.patch
        Optional Tests  
        git revision trunk / 2cb09e9
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6968/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 14s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739719/HADOOP-11965.001.patch Optional Tests   git revision trunk / 2cb09e9 Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6968/console This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        HI Yufei Gu,

        Thanks for the patch. It looks good, only a few nits/suggestions:

        • change DEFAULT_TOP_LINE_NUM to DEFAULT_TOP_NUM_FAILED_TESTS (imagine we may report multiple lines for one test in the future)
        • change num_line to num_failed_tests
        • change error_num to error_count (not to confuse with errno)
        • About "Among x runs examined, top y failed tests ...". The y here need to consider the case that when y is bigger than total number of failed tests. Suggest to change to "Among x runs examined, y tests failed, and the top z failed tests are (<#failedRuns: testName>):", where y is the total number of failed tests, and make the z here MIN(z, y).
        • In non-summary mode, modify the report to "Among x runs examined, all y failed tests are (<#failedRuns: testName>):
          *
          logging.info("\n" + str(error_num) + " errors found, check error details with non summary mode."
          

          Change to

          logging.info("\n" + str(error_num) + " errors found,  you may re-run in non summary mode to see error details."
          

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - HI Yufei Gu , Thanks for the patch. It looks good, only a few nits/suggestions: change DEFAULT_TOP_LINE_NUM to DEFAULT_TOP_NUM_FAILED_TESTS (imagine we may report multiple lines for one test in the future) change num_line to num_failed_tests change error_num to error_count (not to confuse with errno) About "Among x runs examined, top y failed tests ...". The y here need to consider the case that when y is bigger than total number of failed tests. Suggest to change to "Among x runs examined, y tests failed, and the top z failed tests are (<#failedRuns: testName>):", where y is the total number of failed tests, and make the z here MIN(z, y). In non-summary mode, modify the report to "Among x runs examined, all y failed tests are (<#failedRuns: testName>): * logging.info( "\n" + str(error_num) + " errors found, check error details with non summary mode." Change to logging.info( "\n" + str(error_num) + " errors found, you may re-run in non summary mode to see error details." Thanks.
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Yongjun,

        Thanks for the detailed review and all these good suggestion. I will add a new patch after modification.

        Show
        yufeigu Yufei Gu added a comment - Hi Yongjun, Thanks for the detailed review and all these good suggestion. I will add a new patch after modification.
        Hide
        yufeigu Yufei Gu added a comment -

        Hi Yongjun,

        About the "Among x runs examined, top y failed tests ...", it might be better if the program print out "all failed tests ..." if the y is bigger than the total number of failed tests. I've made these changes. Check the HADOOP-11965.002.patch for details.

        Show
        yufeigu Yufei Gu added a comment - Hi Yongjun, About the "Among x runs examined, top y failed tests ...", it might be better if the program print out "all failed tests ..." if the y is bigger than the total number of failed tests. I've made these changes. Check the HADOOP-11965 .002.patch for details.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 35s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12739890/HADOOP-11965.002.patch
        Optional Tests  
        git revision trunk / b039e69
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6972/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739890/HADOOP-11965.002.patch Optional Tests   git revision trunk / b039e69 Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6972/console This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Yufei Gu,

        Thanks for the new rev. LGTM.

        Hi Allen Wittenauer, thank you for reporting the issue. I'm +1 on rev 002. I'd like to commit it some time tomorrow. Would you please let me know if you have any comments?

        Thanks.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Yufei Gu , Thanks for the new rev. LGTM. Hi Allen Wittenauer , thank you for reporting the issue. I'm +1 on rev 002. I'd like to commit it some time tomorrow. Would you please let me know if you have any comments? Thanks.
        Hide
        yzhangal Yongjun Zhang added a comment -

        Hi Yufei Gu,

        I noticed some lines involved in the change exceeded 80 chars per line, would you please create a new rev with this fixed?

        Thanks and sorry for not pointing this out earlier.

        Show
        yzhangal Yongjun Zhang added a comment - Hi Yufei Gu , I noticed some lines involved in the change exceeded 80 chars per line, would you please create a new rev with this fixed? Thanks and sorry for not pointing this out earlier.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 release audit 0m 16s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
            0m 20s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12740231/HADOOP-11965.003.patch
        Optional Tests  
        git revision trunk / cc43288
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6985/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 16s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740231/HADOOP-11965.003.patch Optional Tests   git revision trunk / cc43288 Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6985/console This message was automatically generated.
        Hide
        yzhangal Yongjun Zhang added a comment -

        I committed to trunk and branch-2. Thanks Yufei much for the contribution.

        Show
        yzhangal Yongjun Zhang added a comment - I committed to trunk and branch-2. Thanks Yufei much for the contribution.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8036 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8036/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/determine-flaky-tests-hadoop.py
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8036 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8036/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/determine-flaky-tests-hadoop.py
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #962 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/962/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • dev-support/determine-flaky-tests-hadoop.py
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #962 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/962/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) dev-support/determine-flaky-tests-hadoop.py hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/232/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • dev-support/determine-flaky-tests-hadoop.py
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #232 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/232/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) dev-support/determine-flaky-tests-hadoop.py hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #2160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2160/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • dev-support/determine-flaky-tests-hadoop.py
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2160 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2160/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) dev-support/determine-flaky-tests-hadoop.py hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/221/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/determine-flaky-tests-hadoop.py
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/221/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/determine-flaky-tests-hadoop.py
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/230/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • dev-support/determine-flaky-tests-hadoop.py
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/230/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) hadoop-common-project/hadoop-common/CHANGES.txt dev-support/determine-flaky-tests-hadoop.py
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2178 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2178/)
        HADOOP-11965. determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126)

        • dev-support/determine-flaky-tests-hadoop.py
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2178 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2178/ ) HADOOP-11965 . determine-flaky-tests needs a summary mode. Contributed by Yufei Gu, (yzhang: rev 74351af3b7521b194116258c96270ddaeccd8126) dev-support/determine-flaky-tests-hadoop.py hadoop-common-project/hadoop-common/CHANGES.txt

          People

          • Assignee:
            yufeigu Yufei Gu
            Reporter:
            aw Allen Wittenauer
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development