Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12111 [Umbrella] Split test-patch off into its own TLP
  3. HADOOP-12113

update test-patch branch to latest code

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • HADOOP-12111
    • yetus
    • None
    • Hide
      The following changes have occurred vs. the release currently shipping with Hadoop:


      MAJOR:
      * Documentation

      A first pass at some documentation in markdown format covering beginner, advanced, some architecture/design thoughts, and a glossary covering terms.

      * De-Hadoop-ification / Personalities / Modules

      By far, the largest change was a significant rewrite to create a core engine to enable pulling out most/all of the Hadoop-specific settings into a separate file. This was done in order to support many projects with different source layouts to use effectively the test-patch code base. As a result, the internals now use a concept of modules to dictate what to do where rather than assuming that the root of the source repo is a valid location.

      Project specific logic has been pushed into plugins we call 'personalities'. Personalities allow projects to dictate flags and ordering as well as perform other actions as needed before tests execute. As a result, there are several early personalities for various projects included.

      * Docker support

      Initial support for test-patch to switch to a dedicated docker container during execution, including the ability to provide a custom Dockerfile.

      * Multiple build systems

      Support for ant with the groundwork laid to support more. An early version of a pig personality is included to use this functionality.

      * Multiple JDKs

      It is possible to specify multiple JDKs to run supported tests. Built-in tests that support multiple JDKs are javac, javadoc, and unit.

      * Multiple bug system support

      Preliminary work to enable supporting bug tracking systems beyond Apache's JIRA system has been introduced. A prototype github PR support plugin is bundled. Additionally, large parts of the JIRA support have been pulled out into a separate plugin.

      * API and variable renames/cleanups

      Many, many changes to the names of variables and functions to be less Hadoop-specific and just overall better. As a result, this is a breaking change for any plugins written against the old bits.

      Additionally, many interface stabilities have changed as a result.

      MINOR:

      * A compile+install has been added as the first test step to populate the maven repo with the current code base, skipping over any failed compiles. This will help stabilize build predictability in between runs with different patches.

      * findbugs and ASF license checks have been moved to plug-ins

      * smart-apply-patch has a stray rm fixed.

      * File names of generated test logs, etc, that use the core engine are now consistent across tests.

      * JIRA output is now colored across the entire line, making it easier to spot failing tests.

      * Warn about old versions of shellcheck, because there are some false positives with it.

      * New files are now handled properly.

      * Deleted modules and files should be handled better.

      * Off-by-one errors in the git diff line listing should fix false positives in a few reports.

      * whitespace now errors on tabs in files not named Makefile.

      * xml files are now validated.

      * bash version check has been added to prevent running on older versions.

      * Some tests now default to a summary report rather than listing all results.

      * In order to prepare for eventual split, many defaults have been changed to be yetus-specific rather than Hadoop specific.

      * Eliminated egrep usage in favor of grep -E

      * Many additions/cleanups to report output.

      * Plugins can now add output to the usage message.

      * test-patch temp directory has been renamed.

      * wget errors are more specific now.
      Show
      The following changes have occurred vs. the release currently shipping with Hadoop: MAJOR: * Documentation A first pass at some documentation in markdown format covering beginner, advanced, some architecture/design thoughts, and a glossary covering terms. * De-Hadoop-ification / Personalities / Modules By far, the largest change was a significant rewrite to create a core engine to enable pulling out most/all of the Hadoop-specific settings into a separate file. This was done in order to support many projects with different source layouts to use effectively the test-patch code base. As a result, the internals now use a concept of modules to dictate what to do where rather than assuming that the root of the source repo is a valid location. Project specific logic has been pushed into plugins we call 'personalities'. Personalities allow projects to dictate flags and ordering as well as perform other actions as needed before tests execute. As a result, there are several early personalities for various projects included. * Docker support Initial support for test-patch to switch to a dedicated docker container during execution, including the ability to provide a custom Dockerfile. * Multiple build systems Support for ant with the groundwork laid to support more. An early version of a pig personality is included to use this functionality. * Multiple JDKs It is possible to specify multiple JDKs to run supported tests. Built-in tests that support multiple JDKs are javac, javadoc, and unit. * Multiple bug system support Preliminary work to enable supporting bug tracking systems beyond Apache's JIRA system has been introduced. A prototype github PR support plugin is bundled. Additionally, large parts of the JIRA support have been pulled out into a separate plugin. * API and variable renames/cleanups Many, many changes to the names of variables and functions to be less Hadoop-specific and just overall better. As a result, this is a breaking change for any plugins written against the old bits. Additionally, many interface stabilities have changed as a result. MINOR: * A compile+install has been added as the first test step to populate the maven repo with the current code base, skipping over any failed compiles. This will help stabilize build predictability in between runs with different patches. * findbugs and ASF license checks have been moved to plug-ins * smart-apply-patch has a stray rm fixed. * File names of generated test logs, etc, that use the core engine are now consistent across tests. * JIRA output is now colored across the entire line, making it easier to spot failing tests. * Warn about old versions of shellcheck, because there are some false positives with it. * New files are now handled properly. * Deleted modules and files should be handled better. * Off-by-one errors in the git diff line listing should fix false positives in a few reports. * whitespace now errors on tabs in files not named Makefile. * xml files are now validated. * bash version check has been added to prevent running on older versions. * Some tests now default to a summary report rather than listing all results. * In order to prepare for eventual split, many defaults have been changed to be yetus-specific rather than Hadoop specific. * Eliminated egrep usage in favor of grep -E * Many additions/cleanups to report output. * Plugins can now add output to the usage message. * test-patch temp directory has been renamed. * wget errors are more specific now.

    Description

      sekikn and I have been working on github. We should update the codebase to reflect all of those changes.

      Attachments

        1. HADOOP-12113-HADOOP-12111.patch
          211 kB
          Allen Wittenauer
        2. HADOOP-12113.HADOOP-12111.01.patch
          218 kB
          Allen Wittenauer

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-11923 test-patch whitespace checker doesn't flag new files

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-11915 test-patch.sh should be documented

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-12034 Wrong comment for the filefilter function in test-patch checkstyle plugin

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Test - A new unit, integration or system test. HADOOP-12023 test-patch.sh triggered full hdfs unit tests when it shouldn't

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. HADOOP-12031 test-patch.sh should have an xml plugin

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. HADOOP-7334 test-patch should check for hard tabs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              aw Allen Wittenauer
              aw Allen Wittenauer
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: