Added the following things:
1. ivy dependencies on hadoop so we do not have to package hadoop jars, bin and conf files along with hive. With Ivy, the hadoop version can be downloaded automatically. In order to compile with say 0.18.0 you have to specify
Currently the mirror used for downloads is
This can also be configured using -Dhadoop.mirror="url"
2. Added ant based preprocessing so we can exclude code which is incompatible with a certain version of hadoop. The only file that does this right now is HiveInputFormat.java which excludes validateInput method while compiling with 0.19.*. The exclusion macros are defined in ql/build.xml (we may later move this to build-common, if we see cases of conditional compilation in other parts of the hive code - for now it was convenient for me to reuse gen-java directories in ql/build.xml in order to store the preprocessed code).
3. Fixed certain ordering sensitive tests so that they are not dependent on the different behavior of reducer merge algorithm when the reduce key is random.
4. Another minor change is in HiveConf.java, where in the hadoop conf directory is also picked relative to HADOOP_HOME
With 0.17 and 0.18 the following tests still fail (input16_cc.q, input16.q and input3.q). I will open a separate JIRA to address those as those are related to how we discover user defined serdes and user defined functions from aux.jars.
Also once those are fixed, we can add another target in ant (test-long) so that a transaction can be regressed with all versions of hadoop - and this can be run in hudson to validate submitted patches.