Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3835

hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      It looks like by default yarn is bundling core-site.xml, yarn-site.xml in test artifact of hadoop-yarn-server-resourcemanager which means that any downstream project which uses this a dependency can have a problem in picking up the user supplied/environment supplied core-site.xml, yarn-site.xml

      So we should ideally exclude these .xml files from being bundled into the test-jar. (Similar to YARN-1748)

      I also proactively looked at other YARN modules where this might be happening.

      vamsee-MBP:hadoop-yarn-project vamsee$ find . -name "*-site.xml"
      ./hadoop-yarn/conf/yarn-site.xml
      ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/resources/yarn-site.xml
      ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/test/resources/yarn-site.xml
      ./hadoop-yarn/hadoop-yarn-client/src/test/resources/core-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/core-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/core-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/test-classes/core-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/test-classes/yarn-site.xml
      ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/resources/core-site.xml
      

      And out of these only two modules (hadoop-yarn-server-resourcemanager, hadoop-yarn-server-tests) are building test-jars. In future, if we start building test-jar of other modules, we should exclude these xml files from being bundled.

      1. YARN-3835.patch
        0.9 kB
        Vamsee Yarlagadda

        Issue Links

          Activity

          Hide
          vamsee Vamsee Yarlagadda added a comment -

          This patch will fix the issue on hadoop-yarn-server-resourcemanager module.

          Show
          vamsee Vamsee Yarlagadda added a comment - This patch will fix the issue on hadoop-yarn-server-resourcemanager module.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 43s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 yarn tests 50m 50s Tests passed in hadoop-yarn-server-resourcemanager.
              85m 19s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12740772/YARN-3835.patch
          Optional Tests javadoc javac unit
          git revision trunk / 20c03c9
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8296/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8296/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8296/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 43s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 yarn tests 50m 50s Tests passed in hadoop-yarn-server-resourcemanager.     85m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12740772/YARN-3835.patch Optional Tests javadoc javac unit git revision trunk / 20c03c9 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8296/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8296/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8296/console This message was automatically generated.
          Hide
          vamsee Vamsee Yarlagadda added a comment -

          Manually verified tests.jar to see missing core-site.xml, yarn-site.xml

          Show
          vamsee Vamsee Yarlagadda added a comment - Manually verified tests.jar to see missing core-site.xml, yarn-site.xml
          Hide
          rkanter Robert Kanter added a comment -

          +1

          Show
          rkanter Robert Kanter added a comment - +1
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Vamsee. Committed to trunk and branch-2!

          Show
          rkanter Robert Kanter added a comment - Thanks Vamsee. Committed to trunk and branch-2!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8051 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8051/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8051 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8051/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #967 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/967/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #237 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/237/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2165 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2165/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #226 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/226/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/235/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/)
          YARN-3835. hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/ ) YARN-3835 . hadoop-yarn-server-resourcemanager test package bundles core-site.xml, yarn-site.xml (vamsee via rkanter) (rkanter: rev 99271b762129d78c86f3c9733a24c77962b0b3f7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              vamsee Vamsee Yarlagadda
              Reporter:
              vamsee Vamsee Yarlagadda
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development