Hadoop Common
  1. Hadoop Common
  2. HADOOP-10461

Create an extensible single point entry for of HCFS tests - using RawLocalFileSystem as example impl.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently alot of manual inheritance and stub classes are required in order to run the FileSystemBaseContract and FSMainOperations tests. This means that the HCFS Test suite for any given FileSystem spans many classes, and makes it confusing and cumbersome for anyone to create new tests for a file system, and also very hard to audit and update the tests for a particular file system.

      If each FileSystem had a single point of entry for its tests, we would be much better off.

      There are two ways to do this.

      • Have a folder or package for each file system (i.e. .//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal ) Or
      • Wrap all the HCFS Test implementation classes into a single class (i.e. RawLocalFileSystemHCFSTestSuite.java).

      The purpose of this JIRA is to implement a wrapper class for the RawLocalFileSystem, which serves as a single point of entry to ALL RawLocalFileSystem tests. This will cleanup the RawLocalFileSystem tests while providing other HCFS implementors a template they can use for their own filesystems (i.e. they can just copy the code and customize it however they want).

      For example: This is a sampling of the classes necessary to glue Local FS implementations into the Existing generic FileSystem test classes:

      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
      .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
      

      Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any once can copy that suite to build tests for their own file system.

      Using the "Enclosed" Junit (http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html ) class, we can combine all of the RawLocalFileSystem tests into a single class. That way

      • Can save on boiler plate for setup and declaring the same FileSystem stubs over and over again, making the test "logic" more easy to focus on. For example, here are some repeat file context declarations for ViewFS tests:
        01:42:08 {TRUNK} $ git grep "ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper)"
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java:    fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java:    fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java:    return ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
        01:45:19 {TRUNK} $ git grep " FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);"
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsURIs.java:    FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java:    fcView = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
        hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java:    FileContext fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
        
      • Can easily be adopted for other FileSystems.
      • Can read in System properties to skip certain tests , thus providing support for the type of variability that we know FileSystem tests require. These can then be used to implement the semantics of HADOOP-9361.

      Ideally, we could replace RawLocalFileSystem tests with this injector as a second follow up patch to this, it would reduce the overall amount of code required, probably.

        Issue Links

          Activity

          Hide
          jay vyas added a comment -

          Any initial thoughts on this? I can move forward on a patch , as I already have some code for this.

          Show
          jay vyas added a comment - Any initial thoughts on this? I can move forward on a patch , as I already have some code for this.
          Hide
          jay vyas added a comment -

          The following

          TestLocalFsFCStatistics
          TestLocalFSFileContextCreateMkdir
          TestLocalFSFileContextMainOperations
          TestSymlinkLocalFS
          TestSymlinkLocalFSFileContext
          TestSymlinkLocalFSFileSystem

          All can be merged into a single "TestRawLocalHCFS" suite, from my vantage point.

          Show
          jay vyas added a comment - The following TestLocalFsFCStatistics TestLocalFSFileContextCreateMkdir TestLocalFSFileContextMainOperations TestSymlinkLocalFS TestSymlinkLocalFSFileContext TestSymlinkLocalFSFileSystem All can be merged into a single "TestRawLocalHCFS" suite, from my vantage point.
          Hide
          Steve Loughran added a comment -

          Doesn't the contract-driven tests in HADOOP-9361 do a lot of this?

          Show
          Steve Loughran added a comment - Doesn't the contract-driven tests in HADOOP-9361 do a lot of this?
          Hide
          jay vyas added a comment -

          might be.

          Main thrust is consolidating FSContract/mainOps/symlink into one test a into one controller class which any hcfs can copy and use with their fs plugin.

          I better look and confirm were not overlapping ...

          Show
          jay vyas added a comment - might be. Main thrust is consolidating FSContract/mainOps/symlink into one test a into one controller class which any hcfs can copy and use with their fs plugin. I better look and confirm were not overlapping ...
          Hide
          jay vyas added a comment -

          Steve Loughran I don't see you wrapping all tests into a single HCFS Suite like im attempting in this JIRA... https://gist.github.com/jayunit100/10014317

          BUT... Maybe we should wait until AFTER your implementation of HADOOP-9361 ?

          Show
          jay vyas added a comment - Steve Loughran I don't see you wrapping all tests into a single HCFS Suite like im attempting in this JIRA... https://gist.github.com/jayunit100/10014317 BUT... Maybe we should wait until AFTER your implementation of HADOOP-9361 ?
          Hide
          jay vyas added a comment -

          ping Steve Loughran .... Now that HADOOP-9361 is finalized, do you see value in this single point of entry for HCFS tests?

          Show
          jay vyas added a comment - ping Steve Loughran .... Now that HADOOP-9361 is finalized, do you see value in this single point of entry for HCFS tests?
          Hide
          jay vyas added a comment -

          Shall I close this ? Doesnt seem like much interest in it.
          We have docs for how to extend existing tests here https://wiki.apache.org/hadoop/HCFS/.

          Show
          jay vyas added a comment - Shall I close this ? Doesnt seem like much interest in it. We have docs for how to extend existing tests here https://wiki.apache.org/hadoop/HCFS/ .

            People

            • Assignee:
              Unassigned
              Reporter:
              jay vyas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development