The idea behind initial patch (
This patch expects that patch proposed in
BIGTOP-1341 is applied.
Classess TestCLI.java and FSCmdExecutor.java were changed to be hcfs
compliant and moved from hdfs into hcfs module. The full class name
of TestCLI is org.apache.bigtop.itest.hadoop.hcfs.TestCLI now.
Being in hcfs module TestCLI can be used to test any Hadoop filesystem like
HDFS or GlusterFS (no other filesystem were checked though). That said, HDFS is
still a primary filesystem for TestCLI, which means that you can run TestCLI on
HDFS and it would work without any additional configuration. On the other hand
to run TestCLI on GlusterFS, you need to set several properties I have
introduced (you don't need to this with HDFS because TestCLI has hardcoded HDFS
specific defaults for all of them). TestCLI has it's test cases defined in
Since some test cases are HDFS specific (not applicable to any other Hadoop
fs), I moved those test cases into separate file testHDFSConf.xml and added
TestHDFSCLI into hdfs module (subclass of TestCLI). There are just few test
cases in this category, but I expect that all the dfsadmin ones from
BIGTOP-1334 would end up here.
The idea is that with HDFS, you would run both TestCLI and TestHDFSCLI, while
with GlusterFS (on any other hadoop compatible fs), you would run just TestCLI.
Changes of TestCLI to make it hcfs ready
- HCFS fs.default.name Hack (see TestCLI.java source to more details)
- added property hcfs.namenode.pattern: regexp to match namenode
(sometimes TestCLI expected to see namenode hostname with port number in
the output, but this is not applicable to other filesystems)
- added property hcfs.root.groupname: root group
(root group is defined in Hadoop config dfs.permissions.superusergroup
for HDFS, while it's always just root for GlusterFS)
- added property hcfs.dirsize.pattern: regexp to match expected dir size
(this is because HDFS reports zero for size of directories while GlusterFS
- added property hcfs.scheme: defines fs scheme (eg. "hdfs:", * "glusterfs:")
- removed few HDFS specific cases from testHCFSConf.xml (moved to
Note: for GlusterFS test runs, I use the following values: