[HADOOP-11746] rewrite test-patch.sh

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: build, test
• Hadoop Flags: Incompatible change, Reviewed

Release Note

<!-- markdown -->
* test-patch.sh now has new output that is different than the previous versions
* test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included
* JIRA comments now use much more markup to improve readability
* test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode
* If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version.
* Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests.
* Plugins for checkstyle, shellcheck, and whitespace now execute as necessary.
* New test code for mvn site
* A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process.
* Several new options
  * --resetrepo will put test-patch.sh in destructive mode, similar to a normal Jenkins run
  * --testlist allows one to provide a comma delimited list of test subsystems to forcibly execute
  * --modulelist to provide a comma delimited list of module tests to execute in addition to the ones that are automatically detected
  * --offline mode to attempt to stop connecting to the Internet for certain operations
* test-patch.sh now defaults to the POSIX equivalents on Solaris and Illumos-based operating systems
* shelldocs.py may be used to generate test-patch.sh API information
* FindBugs output is now listed on the JIRA comment
* lots of general code cleanup, including attempts to remove any local state files to reduce potential race conditions
* Some logic to determine if a patch is for a given major branch using several strategies as well as a particular git ref (using git+ref as part of the name).
* Some logic to determine if a patch references a particular JIRA issue.
* Unit tests are only flagged as necessary with native or Java code, since Hadoop has no framework in place yet for other types of unit tests.
* test-patch now exits with a failure status if problems arise trying to do git checkouts. Previously the exit code was success.

• Assignee: Allen Wittenauer
• Reporter: Allen Wittenauer
• Created: 25/Mar/15 03:29
• Updated: 30/Aug/16 01:30
• Resolved: 21/Apr/15 20:45

Description

This code is bad and you should feel bad.

Issue Links

• contains: HADOOP-6186 Enhance checkstyle results by Hudson Hadoop QA to provide a diff
• contains: HADOOP-8714 Jenkins cannot detect download failure
• contains: HADOOP-8875 test-patch.sh doesn't test changes to itself
• contains: HADOOP-9572 Enhance Pre-Commit Admin job to test-patch multiple branches
• contains: HADOOP-11845 two small bugs in shelldocs.py
• contains: HADOOP-7435 Make pre-commit checks run against the correct branch
• contains: HADOOP-11220 Jenkins should verify "mvn site" if the patch contains *.apt.vm changes
• contains: HADOOP-7670 test-patch should not -1 for lack of tests just build changes
• contains: HADOOP-11745 Incorporate ShellCheck static analysis into Jenkins pre-commit builds.
• is depended upon by: HADOOP-11866 increase readability and reliability of checkstyle, shellcheck, and whitespace reports
• is depended upon by: HBASE-13525 Update test-patch to leverage Apache Yetus
• is related to: PHOENIX-1888 [build] add pre-commit scripts
• relates to: HADOOP-11861 test-patch.sh rewrite addendum patch
• relates to: HADOOP-11781 fix race conditions and add URL support to smart-apply-patch.sh
• relates to: HADOOP-11881 test-patch.sh javac result is wildly wrong
• requires: HADOOP-11778 Update checkstyle