Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1762

Allow TestHDFSCLI to be run against a cluster

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes.

      1. HDFS-1762-20.patch
        144 kB
        Tom White
      2. HDFS-1762.hdfs.patch
        141 kB
        Konstantin Boudnik
      3. HDFS-1762.hdfs.patch
        141 kB
        Konstantin Boudnik
      4. HDFS-1762.patch
        142 kB
        Konstantin Boudnik
      5. HDFS-1762.hdfs.patch
        142 kB
        Konstantin Boudnik
      6. HDFS-1762.hdfs.patch
        142 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          This could work by looking for system properties that specify the filesystem and jobtracker, dropping back to the mini clusters if neither were specified.

          Some changes to the tests are needed too. Absolute paths at / should be avoided, so a prefix should be allowed. It might be a good idea to split the tests that require superuser privileges (e.g. the quota tests) into a separate file.

          Show
          Tom White added a comment - This could work by looking for system properties that specify the filesystem and jobtracker, dropping back to the mini clusters if neither were specified. Some changes to the tests are needed too. Absolute paths at / should be avoided, so a prefix should be allowed. It might be a good idea to split the tests that require superuser privileges (e.g. the quota tests) into a separate file.
          Hide
          Konstantin Boudnik added a comment -

          Also, it seems to be necessary to include appropriate XML files into hadoop-tests.jar because otherwise TestCLI won't work at all.

          Show
          Konstantin Boudnik added a comment - Also, it seems to be necessary to include appropriate XML files into hadoop-tests.jar because otherwise TestCLI won't work at all.
          Hide
          Todd Lipcon added a comment -

          I don't think adding the XML files to the test jar is a good idea. We've had that in the past, and since the test jar often ends up on other people's classpaths, and can override conf directories, etc.

          Show
          Todd Lipcon added a comment - I don't think adding the XML files to the test jar is a good idea. We've had that in the past, and since the test jar often ends up on other people's classpaths, and can override conf directories, etc.
          Hide
          Konstantin Boudnik added a comment -

          ... can override conf directories

          we are talking about very special kind of file, actually... So, I am not sure how it can override any configs. But if you're saying there were incidents of the sort I'd rather believe you

          Show
          Konstantin Boudnik added a comment - ... can override conf directories we are talking about very special kind of file, actually... So, I am not sure how it can override any configs. But if you're saying there were incidents of the sort I'd rather believe you
          Hide
          Todd Lipcon added a comment -

          I may have misread. In the past, over in HBase land, we had an hbase-site.xml included in our hbase-test.jar. This meant that if someone accidentally ended up with this on their classpath, they picked up weird configurations that we meant to only apply in tests.

          Putting an XML file that's specific to TestHDFSCLI in the jar seems fine, though.

          Show
          Todd Lipcon added a comment - I may have misread. In the past, over in HBase land, we had an hbase-site.xml included in our hbase-test.jar. This meant that if someone accidentally ended up with this on their classpath, they picked up weird configurations that we meant to only apply in tests. Putting an XML file that's specific to TestHDFSCLI in the jar seems fine, though.
          Hide
          Konstantin Boudnik added a comment -

          Todd, I was talking about testHDFSConf.xml file which actually defined test cases executed by TestCLI class. The risk of 'overriding' in this case is pretty much close to 0. I wouldn't dare to suggest to add any hadoop related config files into test artifact.

          Show
          Konstantin Boudnik added a comment - Todd, I was talking about testHDFSConf.xml file which actually defined test cases executed by TestCLI class. The risk of 'overriding' in this case is pretty much close to 0. I wouldn't dare to suggest to add any hadoop related config files into test artifact.
          Hide
          Todd Lipcon added a comment -

          Ah, sorry for the misunderstanding. Including testHDFSConf.xml seems totally fine to me.

          Show
          Todd Lipcon added a comment - Ah, sorry for the misunderstanding. Including testHDFSConf.xml seems totally fine to me.
          Hide
          Tom White added a comment -

          Here's a 20-based patch.

          Show
          Tom White added a comment - Here's a 20-based patch.
          Hide
          Konstantin Boudnik added a comment -

          These are patches for all three subprojects for 0.22 branch. I will create separate tickets in a bit.
          They should be mostly good for 0.23 except the ant build part which needs to be re-worked for maven.

          Show
          Konstantin Boudnik added a comment - These are patches for all three subprojects for 0.22 branch. I will create separate tickets in a bit. They should be mostly good for 0.23 except the ant build part which needs to be re-worked for maven.
          Hide
          Konstantin Boudnik added a comment -

          Patches above works for local mini-cluster setup without any problem. I will be validating this with a real cluster before committing them.

          Show
          Konstantin Boudnik added a comment - Patches above works for local mini-cluster setup without any problem. I will be validating this with a real cluster before committing them.
          Hide
          Roman Shaposhnik added a comment -

          @Cos, please commit it to .22 the Bigtop validation has passed.

          Show
          Roman Shaposhnik added a comment - @Cos, please commit it to .22 the Bigtop validation has passed.
          Hide
          Konstantin Boudnik added a comment -

          Correcting some regexps to reflect changes in the output formats.

          Show
          Konstantin Boudnik added a comment - Correcting some regexps to reflect changes in the output formats.
          Hide
          Konstantin Boudnik added a comment -

          For complete picture look at the patches associated with HADOOP-7730 and MAPREDUCE-3156

          Show
          Konstantin Boudnik added a comment - For complete picture look at the patches associated with HADOOP-7730 and MAPREDUCE-3156
          Hide
          Konstantin Shvachko added a comment -

          The patch looks good. One thing I would recommend to update regular expressions for host names and paths in the spirit of HDFS-2232 and MAPREDUCE-2304.
          Also TestConfCluster.xml need ASF license header.

          Show
          Konstantin Shvachko added a comment - The patch looks good. One thing I would recommend to update regular expressions for host names and paths in the spirit of HDFS-2232 and MAPREDUCE-2304 . Also TestConfCluster.xml need ASF license header.
          Hide
          Konstantin Boudnik added a comment -

          Added missing license. Also, I'd propose to address any regexp issues separately. The reason for two files was to avoid too wide (e.g. generic) expressions in the first place.

          Show
          Konstantin Boudnik added a comment - Added missing license. Also, I'd propose to address any regexp issues separately. The reason for two files was to avoid too wide (e.g. generic) expressions in the first place.
          Hide
          Konstantin Shvachko added a comment -

          The issues I mentioned before are intended to support host names with "-" (etc.) in them. Some tests (like du will fail on my host as it has a dash in it.

          Show
          Konstantin Shvachko added a comment - The issues I mentioned before are intended to support host names with "-" (etc.) in them. Some tests (like du will fail on my host as it has a dash in it.
          Hide
          Konstantin Shvachko added a comment -

          (like du: )

          Show
          Konstantin Shvachko added a comment - (like du: )
          Hide
          Konstantin Boudnik added a comment -

          makes sense - I will update the patch shortly.

          Show
          Konstantin Boudnik added a comment - makes sense - I will update the patch shortly.
          Hide
          Konstantin Boudnik added a comment -

          Addressing issue for hostnames with '-'

          Show
          Konstantin Boudnik added a comment - Addressing issue for hostnames with '-'
          Hide
          Konstantin Boudnik added a comment -

          Fixing comparators for 4 cases. Now everything passes with hdfs-0.22 cluster.

          Show
          Konstantin Boudnik added a comment - Fixing comparators for 4 cases. Now everything passes with hdfs-0.22 cluster.
          Hide
          Konstantin Shvachko added a comment -

          +1

          Show
          Konstantin Shvachko added a comment - +1
          Hide
          Konstantin Boudnik added a comment -

          I have committed it.

          Show
          Konstantin Boudnik added a comment - I have committed it.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #97 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/97/)
          HDFS-1762. Allow TestHDFSCLI to be run against a cluster. Contributed by Tom White, Konstantin Boudnik.

          cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182116
          Files :

          • /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/hdfs/build.xml
          • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/cli/TestHDFSCLI.java
          • /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/cli/testConfCluster.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #97 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/97/ ) HDFS-1762 . Allow TestHDFSCLI to be run against a cluster. Contributed by Tom White, Konstantin Boudnik. cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1182116 Files : /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt /hadoop/common/branches/branch-0.22/hdfs/build.xml /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/cli/TestHDFSCLI.java /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/cli/testConfCluster.xml

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development