I went through the code of hdfs testaces and it seem to me that there are more
changes needed. I propose the following:
1) Change TestCLI.java (and FSCmdExecutor.java which is used by TestCLI) to use
hadoop executable (/usr/lib/hadoop/bin/hadoop) instead of using
org.apache.hadoop.fs.FsShell directly. The choice of which hadoop filesystem
will be used can be done via setting few system properties. This way we
solve several problems:
- fs specific configuration doesn't need to be included in the TestCLI
code, we assume that these tests are run on properly configured hadoop
- there is zero additional work (besides specification of the properties)
to run this test with any hadoop compatible fs. Eg. with current state, to
make it run with glusterfs, one would have to rewrite the test code and
edit pom.xml for classpath to include jar with glusterfs implementation
(which is reduntant as you already specified this in core-site.xml).
- and last but not least using command line is more appropriate for this
kind of testing anyway
My first attempt how this may look like is attached - it's not a patch, I
changed FSCmdExecutor.java and TestCLI.java - and it's not fully working
2) Most of the other tests are HDFS specific and as such should not be run on
hadoop pools using alternative hadoop compatible filesystems:
On the other hand TestDistCpIntra.groovy, TestFileAppend.groovy and
TestTextSnappy.groovy may be executed even on non HDFS pool, but may
require minor updates - I didn't try this yet though).
To make it work with any HCFS, I would create 2 JUnit groups, one for HDFS
specific tests, the other for HCFS tests - but I'm not sure how this would
fit into current workflow. Any better ideas here?
Does this whole idea make sense for you? What would you change?