[PIG-2189] e2e test harness needs to use Pig as a source of truth

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.10.0
• Component/s: tools
• Hadoop Flags: Reviewed

Release Note

Usage:

To use this feature cd to test/e2e/pig/ and do:

ant -Dpig.harness.old.pig=_path_to_old_pig_ -Dpig.harness.cluster=_path_to_top_cluster_dir_

where _path_to_old_pig_ is the path to where you have installed an older version of Pig. For example if you want to use 0.8.1 as the older version (recommended for current testing of 0.9 and trunk) and you have it installed in your home directory under tools, you would set this to "/homes/you/tools/pig-0.8.1".

and _path_top_top_cluster_dir is the path to where you cluster is installed. This must be the top directory (it should have bin, conf, etc. in it) not the conf directory where the *-site.xml files are. If your hadoop program is installed in "/grids/snoopy/bin/hadoop" then you would pass "/grid/snoopy" for this value.

Running the above will run all of the currently specified end-to-end tests. It takes around 8 hours currently.

If you wish to run just some of the tests you can define -Dtests.to.run=_tests_ where _tests_ is of the form "-t _test_or_group_". _test_or_group_ can be a group name (like Checkin) or a particular test (like Checkin_1). You can put multiple of these in one definition. So to run all the checkin tests and the 3rd multiquery test you would do -Dtests.to.run-"-t Checkin -t MultiQuery_3".

Before you run the tests the first time against a new cluster the harness can generate and deploy into the cluster all of the data you need. To do this do:

ant -Dpig.harness.old.pig=_path_to_old_pig_ -Dpig.harness.cluster=_path_to_top_cluster_dir_ deploy

This will not run the tests, just deploy the data. You can then run the tests.

• Assignee: Alan Gates
• Reporter: Alan Gates
• Created: 23/Jul/11 14:25
• Updated: 26/Apr/12 20:32
• Resolved: 26/Jul/11 02:09

Description

Currently the end-to-end test harness requires a Postgresql database to generate expected results for many tests. This makes it hard to install and use in automated builds and by users who don't already have postgresql installed. As per the proposal put forth at https://cwiki.apache.org/confluence/display/PIG/PigTestProposal , we need to change the test harness to test either against old versions of Pig or against alternate Pig Latin scripts.