Enhancements to gridmix scripts

• Key: HADOOP-2888
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Mukund Madhugiri
• Reporter: Mukund Madhugiri
• Component/s: test
• Affects Version/s: 0.17.0
• Fix Version/s: 0.17.0

Description

I would like to propose enhancements to the gridmix scripts to make it:
1. easier to setup parameters for the test run and data generation (makes it easier to automate the runs using something like hudson)
2. ensure the benchmarks wait until they are completed (makes it easier to automate the runs using something like hudson)

Here are the details:
Ability to override these parameters in gridmix-env

• HADOOP_HOME
• GRID_MIX_HOME
• EXAMPLE_JAR
• APP_JAR
• STREAM_JAR
• GRID_MIX_DATA
• GRID_MIX_PROG

Ability to override these parameters in generateData.sh

• COMPRESSED_DATA_BYTES
• UNCOMPRESSED_DATA_BYTES
• INDIRECT_DATA_BYTES

Ability for the tests submitted to the same cluster to wait until they are done. Changes will be in:

• submissionScripts/monsterQueriesToSameCluster
• submissionScripts/maxentToSameCluster
• submissionScripts/textSortToSameCluster
• submissionScripts/webdataScanToSameCluster
• submissionScripts/webdataSortToSameCluster

Comments

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12376542/gridmix_patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 24 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1843/console

This message is automatically generated.

[Chris Douglas]

While this is being updated, it might make sense to change the defaults away from 0.15.2-dev builds of the test, streaming, and example jars, probably to hadoop-test.jar, hadoop-examples.jar, etc. as in:

# set var only if it has not already been set externally
if [ -z "${EXAMPLE_JAR}" ] ; then
  export EXAMPLE_JAR="${HADOOP_HOME}/hadoop-*-examples.jar"
fi

Also, since the USE_REAL_DATASET change, the comments describing the generated data no longer match the adjacent var; these might be more readable if each were moved within the "real" block.

+1 overall

[Mukund Madhugiri]

Changed the patch based on comments from Chris.

Changes are in:

• gridmix-env
• generateData.sh

[Chris Douglas]

In the section:

+if [ -z "${EXAMPLE_JAR}" ] ; then
+  export EXAMPLE_JAR=${HADOOP_HOME}/hadoop-*-examples.jar
+fi

The exported string should be in quotes so bash will interpret it.

[Mukund Madhugiri]

Missed the quotes. Attaching new patch.

[Chris Douglas]

+1

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377543/gridmix_patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 24 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1932/console

This message is automatically generated.

[Chris Douglas]

I just committed this. Thanks Mukund!

[Hudson]

Integrated in Hadoop-trunk #427 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/427/)