[HBASE-8018] Add "Flaky Testcase Detector" tool into dev-tools

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.98.0, 0.95.2
• Component/s: util
• Hadoop Flags: Reviewed

Release Note

Tool that looks at jenkins builds and identifies flakey tests or tests that are 'invisibles' -- not mentioned in the test listing but that nonetheless cause the build to fail. See the README.md under dev-tools/jenkins-tools for how to build and use.

• Assignee: Jeffrey Zhong
• Reporter: Jeffrey Zhong
• Created: 06/Mar/13 23:47
• Updated: 05/Nov/13 19:21
• Resolved: 11/Aug/13 06:33

Description

jenkins-tools
=============

A tool which pulls test case results from Jenkins server. It displays a union of failed test cases
from the last 15(by default and actual number of jobs can be less depending on availablity) runs
recorded in Jenkins sever and track how each of them are performed for all the last 15 runs(passed,
not run or failed)

Pre-requirement(run under folder ./dev-support/jenkins-tools)
Please download jenkins-client from https://github.com/cosmin/jenkins-client
1) git clone git://github.com/cosmin/jenkins-client.git
2) make sure the dependency jenkins-client version in ./buildstats/pom.xml matches the
downloaded jenkins-client(current value is 0.1.6-SNAPSHOT)

Build command(run under folder jenkins-tools):

       mvn clean package

Usage are:

       java -jar ./buildstats/target/buildstats.jar <Jenkins HTTP URL> <Job Name> [number of last most recent jobs to check]

Sample commands are:

       java -jar ./buildstats/target/buildstats.jar https://builds.apache.org HBase-TRUNK

Sample output(where 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"):

Failed Test Cases Stats 4360 4361 4362 4363 4364 4365 4366 4367 4368 4369
========================================================
org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace 1 1 1 1 1 1 1 1 -1 0
org.apache.hadoop.hbase.migration.testnamespaceupgrade.testrenameusingsnapshots 1 1 1 -1 0 1 1 1 1 1

Skipped Test Cases Stats
======= 4360 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.replication.testreplicationkillmasterrscompressed
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4361 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4362 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4363 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfilessplitrecovery
org.apache.hadoop.hbase.mapreduce.testsecureloadincrementalhfiles
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples
======= 4368 skipped(Or don't have) following test suites =======
org.apache.hadoop.hbase.client.testadmin
org.apache.hadoop.hbase.client.testclonesnapshotfromclient
org.apache.hadoop.hbase.mapreduce.testmapreduceexamples