Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1486

Generalize CLITest structure and interfaces to facilitate upstream adoption (e.g. for web testing)

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: test
    • Labels:
      None

      Description

      HDFS part of HADOOP-7014. HDFS side of TestCLI doesn't require any special changes but needs to be aligned with Common

      1. HDFS-1486.patch
        10 kB
        Konstantin Boudnik
      2. HDFS-1486.patch
        10 kB
        Konstantin Boudnik
      3. HDFS-1486.patch
        3 kB
        Konstantin Boudnik
      4. HDFS-1486.patch
        3 kB
        Konstantin Boudnik
      5. HDFS-1486.patch
        3 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          Corresponding fix.

          Show
          Konstantin Boudnik added a comment - Corresponding fix.
          Hide
          Konstantin Boudnik added a comment -

          Renaming interface as per latest changes of the downstream JIRA.

          Show
          Konstantin Boudnik added a comment - Renaming interface as per latest changes of the downstream JIRA.
          Hide
          Konstantin Boudnik added a comment -

          Well, renamed files were missed from the patch. Fixed.

          Show
          Konstantin Boudnik added a comment - Well, renamed files were missed from the patch. Fixed.
          Hide
          Konstantin Boudnik added a comment -

          Aligning names with last downstream patch

          Show
          Konstantin Boudnik added a comment - Aligning names with last downstream patch
          Hide
          Eli Collins added a comment -

          Hey Cos,

          In CmdFactoryDFS seems like getCommandExecutor should be a method of CLICommand, this way you don't need to check cmd instanceof CLITestData.CLITestCmd, and saves you from having CLITestCmd specific implementation code in the abstract CmdFactoryDFS class? Otherwise looks great.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Cos, In CmdFactoryDFS seems like getCommandExecutor should be a method of CLICommand, this way you don't need to check cmd instanceof CLITestData.CLITestCmd, and saves you from having CLITestCmd specific implementation code in the abstract CmdFactoryDFS class? Otherwise looks great. Thanks, Eli
          Hide
          Konstantin Boudnik added a comment -

          I like the idea! It won't free me from casting CLICommand to CLITestData.CLITestCmd but we won't need to have factory classes whatsoever, I think.

          This will actually require to change HADOOP-7014 first though. I'll publish new patch shortly.

          Show
          Konstantin Boudnik added a comment - I like the idea! It won't free me from casting CLICommand to CLITestData.CLITestCmd but we won't need to have factory classes whatsoever, I think. This will actually require to change HADOOP-7014 first though. I'll publish new patch shortly.
          Hide
          Eli Collins added a comment -

          If you override getCommandExecutor it should save you the cast, ie the appropriate method will be called.

          Show
          Eli Collins added a comment - If you override getCommandExecutor it should save you the cast, ie the appropriate method will be called.
          Hide
          Konstantin Boudnik added a comment -

          Eli, I was looking deeper into this code structure optimisation after our off-line conversation. While the code structure is in apparent need to be refactored/cleaned it seems like it is a much bigger problem than this little generalisation.

          If you suggest that this particular effort is being postponed until the overall code structure is improved - it is fine with me. It is just a bigger task.

          Show
          Konstantin Boudnik added a comment - Eli, I was looking deeper into this code structure optimisation after our off-line conversation. While the code structure is in apparent need to be refactored/cleaned it seems like it is a much bigger problem than this little generalisation. If you suggest that this particular effort is being postponed until the overall code structure is improved - it is fine with me. It is just a bigger task.
          Hide
          Konstantin Boudnik added a comment -

          Changes corresponding to the latest HADOOP-7014. TestStorageRestore is failing because of HDFS-1496

          Show
          Konstantin Boudnik added a comment - Changes corresponding to the latest HADOOP-7014 . TestStorageRestore is failing because of HDFS-1496
          Hide
          Konstantin Boudnik added a comment -

          Rebasing against current trunk

          Show
          Konstantin Boudnik added a comment - Rebasing against current trunk
          Hide
          Konstantin Boudnik added a comment -

          Running commit tests:

          % ant run-commit-test  -Dresolvers=internal
          Buildfile: /home/cos/work/H0.23/git/hdfs/build.xml
          ...
          [ivy:resolve] downloading /home/xxx/.m2/repository/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-SNAPSHOT.jar ...
          [ivy:resolve] ....................... (1401kB)
          [ivy:resolve]   [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (34ms)
          ...
              [junit] Running org.apache.hadoop.cli.TestHDFSCLI
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 53.218 sec
          ...
          checkfailure:
          
          BUILD SUCCESSFUL
          Total time: 16 minutes 47 seconds
          

          As I mentioned in the parent JIRA it is difficult to run test-patch for cross-projects change like this one.

          Show
          Konstantin Boudnik added a comment - Running commit tests: % ant run-commit-test -Dresolvers=internal Buildfile: /home/cos/work/H0.23/git/hdfs/build.xml ... [ivy:resolve] downloading /home/xxx/.m2/repository/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-SNAPSHOT.jar ... [ivy:resolve] ....................... (1401kB) [ivy:resolve] [SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (34ms) ... [junit] Running org.apache.hadoop.cli.TestHDFSCLI [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 53.218 sec ... checkfailure: BUILD SUCCESSFUL Total time: 16 minutes 47 seconds As I mentioned in the parent JIRA it is difficult to run test-patch for cross-projects change like this one.
          Hide
          Konstantin Boudnik added a comment -

          I have also ran full test cycle and see only

              [junit] Tests run: 7, Failures: 0, Errors: 2, Time elapsed: 14.762 sec
              [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED
          

          with java.io.IOException: Too many open files. Apparently isn't caused by the change in question.

          Show
          Konstantin Boudnik added a comment - I have also ran full test cycle and see only [junit] Tests run: 7, Failures: 0, Errors: 2, Time elapsed: 14.762 sec [junit] Test org.apache.hadoop.hdfs.TestFileConcurrentReader FAILED with java.io.IOException: Too many open files . Apparently isn't caused by the change in question.
          Hide
          Konstantin Boudnik added a comment -

          ready to roll.

          Show
          Konstantin Boudnik added a comment - ready to roll.
          Hide
          Konstantin Boudnik added a comment -

          Well, waiting for test-patch does't make sense. Test-patch won't be able to complete because the very first step of the validation will always fail because the compilation will be failing (current HDFS won't compile against new Common because of the API incompatibility). Thus, the testing I have done above seems to be sufficient and the only possible in this situation.

          I am going to commit this patch to fix the build.

          Show
          Konstantin Boudnik added a comment - Well, waiting for test-patch does't make sense. Test-patch won't be able to complete because the very first step of the validation will always fail because the compilation will be failing (current HDFS won't compile against new Common because of the API incompatibility). Thus, the testing I have done above seems to be sufficient and the only possible in this situation. I am going to commit this patch to fix the build.
          Hide
          Konstantin Boudnik added a comment -

          I have just committed this.

          Show
          Konstantin Boudnik added a comment - I have just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #597 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/597/)
          HDFS-1486. Generalize CLITest structure and interfaces to facilitate upstream
          adoption (e.g. for web testing). Contributed by Konstantin Boudnik.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #597 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/597/ ) HDFS-1486 . Generalize CLITest structure and interfaces to facilitate upstream adoption (e.g. for web testing). Contributed by Konstantin Boudnik.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #643 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/ )

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development