Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12098

Remove redundant test dependencies in Hadoop Archives

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      hadoop-mapreduce-client-hs and hadoop-mapreduce-client-jobclient(test jar) dependencies are not required in Hadoop archives

      1. HADOOP-12098.001.patch
        1 kB
        Varun Saxena
      2. HADOOP-12098.02.patch
        1.0 kB
        Varun Saxena

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Patch for some reason is not being picked up by Jenkins. Resubmitting again.

        Show
        varun_saxena Varun Saxena added a comment - Patch for some reason is not being picked up by Jenkins. Resubmitting again.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 39s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 53s There were no new javac warning messages.
        +1 javadoc 10m 1s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 tools/hadoop tests 0m 30s Tests failed in hadoop-archives.
            36m 38s  



        Reason Tests
        Failed unit tests hadoop.tools.TestHadoopArchives



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742063/HADOOP-12098.001.patch
        Optional Tests javadoc javac unit
        git revision trunk / 8ef07f7
        hadoop-archives test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/artifact/patchprocess/testrun_hadoop-archives.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 53s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 tools/hadoop tests 0m 30s Tests failed in hadoop-archives.     36m 38s   Reason Tests Failed unit tests hadoop.tools.TestHadoopArchives Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742063/HADOOP-12098.001.patch Optional Tests javadoc javac unit git revision trunk / 8ef07f7 hadoop-archives test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/artifact/patchprocess/testrun_hadoop-archives.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7050/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Tests fail due to runtime dependence on jobclient main jar. The test jar and hs jars are still unnecessary.
        The runtime dependence comes due to subclasses of ClientProtocolProvider classes being loaded at runtime

        ClientProtocolProvider exists in hadoop-mapreduce-core.
        LocalClientProtocolProvider and associated classes such as LocalJobRunner exist in hadoop-mapreduce-client-common
        YarnClientProtocolProvider and associated classes such as YarnRunner exist in hadoop-mapreduce-jobclient.
        Any reason why these classes have been kept this way ? All can move to hadoop-mapreduce-client-core IMHO. This creates unnecessary dependencies.

        hadoop-mapreduce-client-jobclient seems to be having hardly any main classes now. Is it even required as a separate submodule ?

        Show
        varun_saxena Varun Saxena added a comment - Tests fail due to runtime dependence on jobclient main jar. The test jar and hs jars are still unnecessary. The runtime dependence comes due to subclasses of ClientProtocolProvider classes being loaded at runtime ClientProtocolProvider exists in hadoop-mapreduce-core . LocalClientProtocolProvider and associated classes such as LocalJobRunner exist in hadoop-mapreduce-client-common YarnClientProtocolProvider and associated classes such as YarnRunner exist in hadoop-mapreduce-jobclient . Any reason why these classes have been kept this way ? All can move to hadoop-mapreduce-client-core IMHO. This creates unnecessary dependencies. hadoop-mapreduce-client-jobclient seems to be having hardly any main classes now. Is it even required as a separate submodule ?
        Hide
        varun_saxena Varun Saxena added a comment -

        Anyways, there is quite a bit of cross dependency in classes used so not going to touch these mapreduce modules.
        Removing dependencies from archives for now.

        Show
        varun_saxena Varun Saxena added a comment - Anyways, there is quite a bit of cross dependency in classes used so not going to touch these mapreduce modules. Removing dependencies from archives for now.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 44s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 tools/hadoop tests 0m 50s Tests passed in hadoop-archives.
            35m 15s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742306/HADOOP-12098.02.patch
        Optional Tests javadoc javac unit
        git revision trunk / fe6c1bd
        hadoop-archives test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/artifact/patchprocess/testrun_hadoop-archives.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 44s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 tools/hadoop tests 0m 50s Tests passed in hadoop-archives.     35m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742306/HADOOP-12098.02.patch Optional Tests javadoc javac unit git revision trunk / fe6c1bd hadoop-archives test log https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/artifact/patchprocess/testrun_hadoop-archives.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7070/console This message was automatically generated.
        Hide
        wheat9 Haohui Mai added a comment -

        Since the patch only remove dependencies for tests it should be fine from a compatibility prospective. +1. Committing.

        Show
        wheat9 Haohui Mai added a comment - Since the patch only remove dependencies for tests it should be fine from a compatibility prospective. +1. Committing.
        Hide
        wheat9 Haohui Mai added a comment -

        I've committed the patch to trunk and branch-2. Thanks Varun Saxena for the contribution.

        Show
        wheat9 Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Varun Saxena for the contribution.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8849 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8849/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-archives/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8849 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8849/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-archives/pom.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2638 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2638/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-tools/hadoop-archives/pom.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2638 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2638/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-tools/hadoop-archives/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #698 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/698/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-archives/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #698 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/698/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-archives/pom.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #709 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/709/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-archives/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #709 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/709/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-archives/pom.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1434 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1434/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-archives/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1434 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1434/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-archives/pom.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2567/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-tools/hadoop-archives/pom.xml
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2567 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2567/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-tools/hadoop-archives/pom.xml hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #629 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/629/)
        HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-tools/hadoop-archives/pom.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #629 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/629/ ) HADOOP-12098 . Remove redundant test dependencies in Hadoop Archives. (wheat9: rev 0d27917042bb84c44c70793295929653017b9eca) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-archives/pom.xml
        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Haohui Mai for the commit.

        Show
        varun_saxena Varun Saxena added a comment - Thanks Haohui Mai for the commit.

          People

          • Assignee:
            varun_saxena Varun Saxena
            Reporter:
            varun_saxena Varun Saxena
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development