Hadoop Common
  1. Hadoop Common
  2. HADOOP-5135

Separate the core, hdfs and mapred junit tests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To support splitting of projects, the tests should be separated into different directories.

      1. 5135_v3.tar
        70 kB
        Sharad Agarwal
      2. 5135_v2.tar
        70 kB
        Sharad Agarwal
      3. 5135_v1.tar
        50 kB
        Sharad Agarwal

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Sharad Agarwal added a comment -

          In the current folder structure, the separation could look like as follows:
          src/test/core - testcases only related to core apis
          src/test/hdfs - testcases dependent on hdfs apis. May depend on core apis/tests as well.
          src/test/mapred - testcases dependent on mapred apis. May depend on hdfs apis/tests as well.
          src/test/hdfs-integration (better name ?)- hdfs tests which depend on mapred for execution. for example benchmark tests NNBench.

          There could be separate targets for all the above say - test-core, test-hdfs, test-mapred, test-hdfs-integration in the current build.xml

          When the project split, the hdfs subproject need not install mapred. It only needs mapred if wants to run test-hdfs-integration. mapred would have to install hdfs to run its tests as most of the tests run with hdfs.

          Show
          Sharad Agarwal added a comment - In the current folder structure, the separation could look like as follows: src/test/core - testcases only related to core apis src/test/hdfs - testcases dependent on hdfs apis. May depend on core apis/tests as well. src/test/mapred - testcases dependent on mapred apis. May depend on hdfs apis/tests as well. src/test/hdfs-integration (better name ?)- hdfs tests which depend on mapred for execution. for example benchmark tests NNBench. There could be separate targets for all the above say - test-core, test-hdfs, test-mapred, test-hdfs-integration in the current build.xml When the project split, the hdfs subproject need not install mapred. It only needs mapred if wants to run test-hdfs-integration. mapred would have to install hdfs to run its tests as most of the tests run with hdfs.
          Hide
          Sharad Agarwal added a comment -

          untar the attached file in hadoop home dir. Run the 5135_script.sh. This script creates the new test source folders and does the svn mv of the respective files. It also applies the patch to build.xml in order to run the core, hdfs, mapred tests independently.
          Things remaining:
          There are couple of test classes which cover both mapred and hdfs functionality like TestCLI, AllTestDriver. These I think should be split.
          The target for building test jar needs to be fixed. Separate test jars to be built.

          Show
          Sharad Agarwal added a comment - untar the attached file in hadoop home dir. Run the 5135_script.sh. This script creates the new test source folders and does the svn mv of the respective files. It also applies the patch to build.xml in order to run the core, hdfs, mapred tests independently. Things remaining: There are couple of test classes which cover both mapred and hdfs functionality like TestCLI, AllTestDriver. These I think should be split. The target for building test jar needs to be fixed. Separate test jars to be built.
          Hide
          V.Karthikeyan added a comment -

          The patch is breaking the trunk and creates build.xml.reg file. Can you please update the patch?

          Show
          V.Karthikeyan added a comment - The patch is breaking the trunk and creates build.xml.reg file. Can you please update the patch?
          Hide
          Sharad Agarwal added a comment -

          Updated to trunk.

          Show
          Sharad Agarwal added a comment - Updated to trunk.
          Hide
          Giridharan Kesavan added a comment -

          build.xml changes in the patch looks good to me. Except for the test-core target.

          test-patch.sh uses test-core and test-contrib target as the target for executing the junit test for the hadoop core and contrib resp.

          Please change the ant target names in this patch accordingly.

          Tnx!

          Show
          Giridharan Kesavan added a comment - build.xml changes in the patch looks good to me. Except for the test-core target. test-patch.sh uses test-core and test-contrib target as the target for executing the junit test for the hadoop core and contrib resp. Please change the ant target names in this patch accordingly. Tnx!
          Hide
          Sharad Agarwal added a comment -

          Updated the patch incorporating Giri's comments.
          Changed the target name to "run-test-core". "test-core" continues to run all non-contrib tests.

          Show
          Sharad Agarwal added a comment - Updated the patch incorporating Giri's comments. Changed the target name to "run-test-core". "test-core" continues to run all non-contrib tests.
          Hide
          Sharad Agarwal added a comment -

          All tests passed on my machine.

          Show
          Sharad Agarwal added a comment - All tests passed on my machine.
          Hide
          Sharad Agarwal added a comment -

          All earlier targets continue to work the same way. Newly added targets for each components are:
          run-test-core
          run-test-hdfs and run-test-hdfs-with-mr
          run-test-mapred

          Show
          Sharad Agarwal added a comment - All earlier targets continue to work the same way. Newly added targets for each components are: run-test-core run-test-hdfs and run-test-hdfs-with-mr run-test-mapred
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Sharad!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Sharad!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk #830 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/)
          . Trying to delete the empty src/test/org directory
          . Splits the tests into different directories based on the package. Four new test targets have been defined - run-test-core, run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr. Contributed by Sharad Agarwal.

          Show
          Hudson added a comment - Integrated in Hadoop-trunk #830 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/ ) . Trying to delete the empty src/test/org directory . Splits the tests into different directories based on the package. Four new test targets have been defined - run-test-core, run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr. Contributed by Sharad Agarwal.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The following tests are under the org.apache.hadoop.fs package but were moved to hdfs sub-directory:

          ./org/apache/hadoop/fs/ftp/TestFTPFileSystem.java
          ./org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java
          ./org/apache/hadoop/fs/permission/TestStickyBit.java
          ./org/apache/hadoop/fs/TestGlobPaths.java
          ./org/apache/hadoop/fs/TestUrlStreamHandler.java
          
          • Some of them are not related to hdfs, e.g. TestFTPFileSystem. These files should be moved out from hdfs and should not use hdfs codes.
          • Some of them are testing hdfs features, e.g. TestStickyBit. They should be defined under org.apache.hadoop.hdfs package.
          Show
          Tsz Wo Nicholas Sze added a comment - The following tests are under the org.apache.hadoop.fs package but were moved to hdfs sub-directory: ./org/apache/hadoop/fs/ftp/TestFTPFileSystem.java ./org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java ./org/apache/hadoop/fs/permission/TestStickyBit.java ./org/apache/hadoop/fs/TestGlobPaths.java ./org/apache/hadoop/fs/TestUrlStreamHandler.java Some of them are not related to hdfs, e.g. TestFTPFileSystem. These files should be moved out from hdfs and should not use hdfs codes. Some of them are testing hdfs features, e.g. TestStickyBit. They should be defined under org.apache.hadoop.hdfs package.
          Hide
          Sharad Agarwal added a comment -

          @Tsz I agree that tests need to be refactored as you have suggested. Actually the split has happened based on the code dependency and not on the package. The commit message perhaps is misleading. This issue mostly dealt with moving the tests into different categories. Refactoring individual tests to belong to right project can be subsequent issues.

          Show
          Sharad Agarwal added a comment - @Tsz I agree that tests need to be refactored as you have suggested. Actually the split has happened based on the code dependency and not on the package. The commit message perhaps is misleading. This issue mostly dealt with moving the tests into different categories. Refactoring individual tests to belong to right project can be subsequent issues.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Sharad, thanks for your comment. I will file issues for the refactoring.

          Show
          Tsz Wo Nicholas Sze added a comment - Sharad, thanks for your comment. I will file issues for the refactoring.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Filed HADOOP-5962.

          Show
          Tsz Wo Nicholas Sze added a comment - Filed HADOOP-5962 .

            People

            • Assignee:
              Sharad Agarwal
              Reporter:
              Sharad Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development