Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      HADOOP-10101 upgraded the guava version for 3.x to 21.

      Guava is broadly used by Java projects that consume our artifacts. Unfortunately, these projects also consume our private artifacts like hadoop-hdfs. They also are unlikely on the new shaded client introduced by HADOOP-11804, currently only available in 3.0.0-alpha2.

      We should shade Guava everywhere to proactively avoid breaking downstreams. This isn't a requirement for all dependency upgrades, but it's necessary for known-bad dependencies like Guava.

      1. HADOOP-14284.012.patch
        1.18 MB
        Tsuyoshi Ozawa
      2. HADOOP-14284.010.patch
        1.13 MB
        Tsuyoshi Ozawa
      3. HADOOP-14284.007.patch
        1.13 MB
        Tsuyoshi Ozawa
      4. HADOOP-14284.004.patch
        40 kB
        Tsuyoshi Ozawa
      5. HADOOP-14284.002.patch
        28 kB
        Tsuyoshi Ozawa
      6. HADOOP-14238.pre001.patch
        22 kB
        Tsuyoshi Ozawa

        Issue Links

          Activity

          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          This is a WIP patch which includes lots redundant parts and useless changes. However, it helps to clarify an approach I'm trying, so attached it here. Based on discussion on HADOOP-13363 with Stack, I'm trying to add hadoop-shaded-3rdparty-libraries.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited This is a WIP patch which includes lots redundant parts and useless changes. However, it helps to clarify an approach I'm trying, so attached it here. Based on discussion on HADOOP-13363 with Stack, I'm trying to add hadoop-shaded-3rdparty-libraries.
          Hide
          andrew.wang Andrew Wang added a comment -

          This is pretty similar to something Sean Busbey and I discussed, I like it!

          One nit suggestion: simplify name to hadoop-shaded-thirdparty?

          Show
          andrew.wang Andrew Wang added a comment - This is pretty similar to something Sean Busbey and I discussed, I like it! One nit suggestion: simplify name to hadoop-shaded-thirdparty?
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Andrew for the feedback! Okay, I will attach a patch including your suggestion, renaming the package name to hadoop-shaded-thirdparty.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Andrew for the feedback! Okay, I will attach a patch including your suggestion, renaming the package name to hadoop-shaded-thirdparty.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 50s Maven dependency ordering for branch
          +1 mvninstall 14m 34s trunk passed
          +1 compile 16m 51s trunk passed
          +1 mvnsite 9m 52s trunk passed
          +1 mvneclipse 1m 6s trunk passed
          +1 javadoc 4m 59s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 30m 26s the patch passed
          +1 compile 13m 28s the patch passed
          +1 javac 13m 28s the patch passed
          +1 mvnsite 9m 0s the patch passed
          +1 mvneclipse 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 34s The patch has no ill-formed XML file.
          +1 javadoc 4m 23s the patch passed
          -1 unit 18m 42s root in the patch failed.
          -1 asflicense 0m 32s The patch generated 2 ASF License warnings.
          128m 45s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14284
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862436/HADOOP-14238.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 048c134694df 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e7167e4
          Default Java 1.8.0_121
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-azure hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 50s Maven dependency ordering for branch +1 mvninstall 14m 34s trunk passed +1 compile 16m 51s trunk passed +1 mvnsite 9m 52s trunk passed +1 mvneclipse 1m 6s trunk passed +1 javadoc 4m 59s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 30m 26s the patch passed +1 compile 13m 28s the patch passed +1 javac 13m 28s the patch passed +1 mvnsite 9m 0s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 34s The patch has no ill-formed XML file. +1 javadoc 4m 23s the patch passed -1 unit 18m 42s root in the patch failed. -1 asflicense 0m 32s The patch generated 2 ASF License warnings. 128m 45s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14284 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862436/HADOOP-14238.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 048c134694df 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e7167e4 Default Java 1.8.0_121 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-azure hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12058/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          I presume we end up using our project wide LICENSE/NOTICE in this jar?

          53	      <plugin>
          54	        <groupId>org.apache.maven.plugins</groupId>
          55	        <artifactId>maven-shade-plugin</artifactId>
          56	        <executions>
          57	          <execution>
          58	            <goals>
          59	              <goal>shade</goal>
          60	            </goals>
          61	            <phase>package</phase>
          62	            <configuration>
          63	              <createDependencyReducedPom>false</createDependencyReducedPom>
          64	              <relocations>
          

          Don't we need to have a dependency reduced pom in order to avoid having Guava show up as a transitive dependency?

          
          75	              <filters>
          76	                <filter>
          77	                  <artifact>com.google.guava:guava</artifact>
          78	                  <excludes>
          79	                    <exclude>META-INF/**</exclude>
          80	                  </excludes>
          81	                </filter>
          82	              </filters>
          

          What are we trying to exclude here? I don't see much of consequence in the guava jar.

          Show
          busbey Sean Busbey added a comment - I presume we end up using our project wide LICENSE/NOTICE in this jar? 53 <plugin> 54 <groupId>org.apache.maven.plugins</groupId> 55 <artifactId>maven-shade-plugin</artifactId> 56 <executions> 57 <execution> 58 <goals> 59 <goal>shade</goal> 60 </goals> 61 <phase> package </phase> 62 <configuration> 63 <createDependencyReducedPom> false </createDependencyReducedPom> 64 <relocations> Don't we need to have a dependency reduced pom in order to avoid having Guava show up as a transitive dependency? 75 <filters> 76 <filter> 77 <artifact>com.google.guava:guava</artifact> 78 <excludes> 79 <exclude>META-INF/**</exclude> 80 </excludes> 81 </filter> 82 </filters> What are we trying to exclude here? I don't see much of consequence in the guava jar.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Thanks Sean for your review.

          I presume we end up using our project wide LICENSE/NOTICE in this jar?

          What are we trying to exclude here? I don't see much of consequence in the guava jar.

          I think we shouldn't include our L&N files here since we only use original Guava. Hence, my intension to exclude META-INF/** is to remove our L&N files and to include L file which is included in original Guava. Precisely, the L file is COPYING file in Guava's repository. The change of excluding META-INF/**, however, was not correct solution as you pointed out - it doesn't work. Thanks for pointing out. Instead of that, I added dummy maven-remote-resources-plugin's process-resources section to remove our original L&N files.

          Don't we need to have a dependency reduced pom in order to avoid having Guava show up as a transitive dependency?

          I addressed to make the flag true in v4 patch.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Thanks Sean for your review. I presume we end up using our project wide LICENSE/NOTICE in this jar? What are we trying to exclude here? I don't see much of consequence in the guava jar. I think we shouldn't include our L&N files here since we only use original Guava. Hence, my intension to exclude META-INF/** is to remove our L&N files and to include L file which is included in original Guava. Precisely, the L file is COPYING file in Guava's repository. The change of excluding META-INF/**, however, was not correct solution as you pointed out - it doesn't work. Thanks for pointing out. Instead of that, I added dummy maven-remote-resources-plugin's process-resources section to remove our original L&N files. Don't we need to have a dependency reduced pom in order to avoid having Guava show up as a transitive dependency? I addressed to make the flag true in v4 patch.
          Hide
          busbey Sean Busbey added a comment -

          Sorry, late on these

          diff --git hadoop-client-modules/hadoop-client-api/pom.xml hadoop-client-modules/hadoop-client-api/pom.xml
          164	                        <!-- Exclude config keys for Guava that is shaded in hadoop-common -->
          165	                        <exclude>com/google/common/*</exclude>
          
          diff --git hadoop-client-modules/hadoop-client-minicluster/pom.xml hadoop-client-modules/hadoop-client-minicluster/pom.xml
          691	                        <!-- Exclude config keys for Guava that is shaded in hadoop-common -->
          692	                        <exclude>com/google/common/*</exclude>
          
          diff --git hadoop-client-modules/hadoop-client-runtime/pom.xml hadoop-client-modules/hadoop-client-runtime/pom.xml
          237	                        <!-- Exclude config keys for Guava that is shaded in hadoop-common -->
          238	                        <exclude>com/google/common/*</exclude>
          

          Won't all of these have been rewritten by the third party module? Otherwise mustn't whatever is in hadoop-common be referring to a non-relocated Guava?

          Related to the above, I don't see where the modules that now depend on shaded-third-party are changed to reference the relocated version of Guava. We can do it either in the source code or in the build, but I don't see us doing either ATM.

          For example, apache avro does this by making a temp jar with non-relocated classes they reference, then doing the relocation for both the classes and their references in the user facing module. Apache HBase does the update-the-source approach for their relocated Google Protobuf. (There's not a simple thing to point at for that example, unfortunately. the README for the module that does the relocation is the closest, since it at least has pointers to what's going on and the commits involved.)

          Show
          busbey Sean Busbey added a comment - Sorry, late on these diff --git hadoop-client-modules/hadoop-client-api/pom.xml hadoop-client-modules/hadoop-client-api/pom.xml 164 <!-- Exclude config keys for Guava that is shaded in hadoop-common --> 165 <exclude>com/google/common/*</exclude> diff --git hadoop-client-modules/hadoop-client-minicluster/pom.xml hadoop-client-modules/hadoop-client-minicluster/pom.xml 691 <!-- Exclude config keys for Guava that is shaded in hadoop-common --> 692 <exclude>com/google/common/*</exclude> diff --git hadoop-client-modules/hadoop-client-runtime/pom.xml hadoop-client-modules/hadoop-client-runtime/pom.xml 237 <!-- Exclude config keys for Guava that is shaded in hadoop-common --> 238 <exclude>com/google/common/*</exclude> Won't all of these have been rewritten by the third party module? Otherwise mustn't whatever is in hadoop-common be referring to a non-relocated Guava? Related to the above, I don't see where the modules that now depend on shaded-third-party are changed to reference the relocated version of Guava. We can do it either in the source code or in the build, but I don't see us doing either ATM. For example, apache avro does this by making a temp jar with non-relocated classes they reference, then doing the relocation for both the classes and their references in the user facing module . Apache HBase does the update-the-source approach for their relocated Google Protobuf. (There's not a simple thing to point at for that example, unfortunately. the README for the module that does the relocation is the closest, since it at least has pointers to what's going on and the commits involved.)
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Sean! I've suspected the possibility of not using third-party jar. Hence, the information of Avro and HBase is helpful for me. Let me check the documentation for a while.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Sean! I've suspected the possibility of not using third-party jar. Hence, the information of Avro and HBase is helpful for me. Let me check the documentation for a while.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 15m 28s trunk passed
          +1 compile 17m 38s trunk passed
          +1 mvnsite 11m 32s trunk passed
          +1 mvneclipse 1m 57s trunk passed
          +1 javadoc 5m 40s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 34m 15s the patch passed
          +1 compile 14m 11s the patch passed
          +1 javac 14m 11s the patch passed
          +1 mvnsite 9m 27s the patch passed
          +1 mvneclipse 1m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 37s The patch has no ill-formed XML file.
          +1 javadoc 4m 23s the patch passed
          -1 unit 16m 50s root in the patch failed.
          -1 asflicense 0m 32s The patch generated 2 ASF License warnings.
          136m 5s



          Reason Tests
          Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem
            hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-14284
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862683/HADOOP-14284.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux b877768c7b40 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 443aa51
          Default Java 1.8.0_121
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 15m 28s trunk passed +1 compile 17m 38s trunk passed +1 mvnsite 11m 32s trunk passed +1 mvneclipse 1m 57s trunk passed +1 javadoc 5m 40s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 34m 15s the patch passed +1 compile 14m 11s the patch passed +1 javac 14m 11s the patch passed +1 mvnsite 9m 27s the patch passed +1 mvneclipse 1m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 37s The patch has no ill-formed XML file. +1 javadoc 4m 23s the patch passed -1 unit 16m 50s root in the patch failed. -1 asflicense 0m 32s The patch generated 2 ASF License warnings. 136m 5s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem   hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14284 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862683/HADOOP-14284.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux b877768c7b40 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 443aa51 Default Java 1.8.0_121 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12072/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          I prefer the way which HBase choose, because it enable us to debug code easily without uploading jar. Let me know if you have any idea.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited I prefer the way which HBase choose, because it enable us to debug code easily without uploading jar. Let me know if you have any idea.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Rebasing on trunk.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Rebasing on trunk.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Some changes in v6 were not intended, so fixing it in v7.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Some changes in v6 were not intended, so fixing it in v7.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user oza opened a pull request:

          https://github.com/apache/hadoop/pull/210

          HADOOP-14284. Shade Guava everywhere.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/oza/hadoop HADOOP-14284

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/210.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #210


          commit 8a38c006ecc25bec26d40da4abda03cdccff85f5
          Author: Tsuyoshi Ozawa <ozawa@apache.org>
          Date: 2017-04-06T10:49:42Z

          HADOOP-14284. Shade Guava everywhere.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user oza opened a pull request: https://github.com/apache/hadoop/pull/210 HADOOP-14284 . Shade Guava everywhere. You can merge this pull request into a Git repository by running: $ git pull https://github.com/oza/hadoop HADOOP-14284 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/210.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #210 commit 8a38c006ecc25bec26d40da4abda03cdccff85f5 Author: Tsuyoshi Ozawa <ozawa@apache.org> Date: 2017-04-06T10:49:42Z HADOOP-14284 . Shade Guava everywhere.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          v7 patch includes following changes:

          • Created hadoop-shaded-thirdparty which includes shaded guava. It, however, doesn't include com.google.common.base.Function, Predicate, and TypeToken. It's because the classes are required to use Apache Curator: https://cwiki.apache.org/confluence/display/CURATOR/TN13
          • Changed namespaces of Guava to use shaded Guava under org.apache.hadoop.com.google.common. This change subsume what HADOOP-14238 is addressing.
          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited v7 patch includes following changes: Created hadoop-shaded-thirdparty which includes shaded guava. It, however, doesn't include com.google.common.base.Function, Predicate, and TypeToken. It's because the classes are required to use Apache Curator: https://cwiki.apache.org/confluence/display/CURATOR/TN13 Changed namespaces of Guava to use shaded Guava under org.apache.hadoop.com.google.common. This change subsume what HADOOP-14238 is addressing.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Tsuyoshi Ozawa for the very tough work!
          I tried the v7 patch and mvn install -DskipTests failed in hadoop-client-check-invariants module:

          [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
          Duplicate classes found:
          
            Found in:
              org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha3-SNAPSHOT:compile
              org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha3-SNAPSHOT:compile
          

          The failure occurs because both the two module has shaded guava. Would you remove shaded guava from hadoop-client-runtime module?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Tsuyoshi Ozawa for the very tough work! I tried the v7 patch and mvn install -DskipTests failed in hadoop-client-check-invariants module: [WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message: Duplicate classes found: Found in: org.apache.hadoop:hadoop-client-runtime:jar:3.0.0-alpha3-SNAPSHOT:compile org.apache.hadoop:hadoop-client-api:jar:3.0.0-alpha3-SNAPSHOT:compile The failure occurs because both the two module has shaded guava. Would you remove shaded guava from hadoop-client-runtime module?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The failure can be fixed by the following setting

          hadoop-client-modules/hadoop-client-runtime/pom.xml
                      <executions>
                        <execution>
                          <phase>package</phase>
                          <goals>
                            <goal>shade</goal>
                          </goals>
                          <configuration>
                            <artifactSet>
                              <excludes>
                                <!-- We need a filter that matches just those things that aer included in the api jar -->
                                <exclude>org.apache.hadoop:hadoop-client-api</exclude>
          +                     <exclude>com.google.guava:guava</exclude>
          
          Show
          ajisakaa Akira Ajisaka added a comment - The failure can be fixed by the following setting hadoop-client-modules/hadoop-client-runtime/pom.xml <executions> <execution> <phase> package </phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <!-- We need a filter that matches just those things that aer included in the api jar --> <exclude>org.apache.hadoop:hadoop-client-api</exclude> + <exclude>com.google.guava:guava</exclude>
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 322 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 52s trunk passed
          +1 compile 15m 21s trunk passed
          +1 checkstyle 8m 30s trunk passed
          +1 mvnsite 9m 29s trunk passed
          +1 mvneclipse 1m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 26m 6s trunk passed
          +1 javadoc 4m 25s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          -1 mvninstall 14m 59s root in the patch failed.
          -1 compile 0m 32s root in the patch failed.
          -1 javac 0m 32s root in the patch failed.
          -0 checkstyle 8m 3s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263)
          -1 mvnsite 0m 25s root in the patch failed.
          -1 mvneclipse 1m 35s root in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 44s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 33m 7s the patch passed
          -1 javadoc 3m 0s root in the patch failed.
          -1 unit 7m 37s root in the patch failed.
          -1 asflicense 0m 40s The patch generated 2 ASF License warnings.
          214m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-14284
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862886/HADOOP-14284.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9e5fe710ea1f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f050afb
          Default Java 1.8.0_121
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvnsite-root.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvneclipse-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 322 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 52s trunk passed +1 compile 15m 21s trunk passed +1 checkstyle 8m 30s trunk passed +1 mvnsite 9m 29s trunk passed +1 mvneclipse 1m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 26m 6s trunk passed +1 javadoc 4m 25s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch -1 mvninstall 14m 59s root in the patch failed. -1 compile 0m 32s root in the patch failed. -1 javac 0m 32s root in the patch failed. -0 checkstyle 8m 3s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263) -1 mvnsite 0m 25s root in the patch failed. -1 mvneclipse 1m 35s root in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 44s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 33m 7s the patch passed -1 javadoc 3m 0s root in the patch failed. -1 unit 7m 37s root in the patch failed. -1 asflicense 0m 40s The patch generated 2 ASF License warnings. 214m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14284 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862886/HADOOP-14284.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9e5fe710ea1f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f050afb Default Java 1.8.0_121 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvnsite-root.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-mvneclipse-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12082/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 40s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 322 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 15m 11s trunk passed
          +1 compile 17m 14s trunk passed
          +1 checkstyle 7m 54s trunk passed
          +1 mvnsite 10m 25s trunk passed
          +1 mvneclipse 1m 43s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 25m 54s trunk passed
          +1 javadoc 4m 32s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          -1 mvninstall 13m 59s root in the patch failed.
          -1 compile 0m 29s root in the patch failed.
          -1 javac 0m 29s root in the patch failed.
          -0 checkstyle 8m 6s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263)
          -1 mvnsite 0m 24s root in the patch failed.
          -1 mvneclipse 1m 35s root in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 38s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
          +1 findbugs 30m 11s the patch passed
          -1 javadoc 2m 51s root in the patch failed.
          -1 unit 7m 12s root in the patch failed.
          -1 asflicense 0m 38s The patch generated 2 ASF License warnings.
          212m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-14284
          GITHUB PR https://github.com/apache/hadoop/pull/210
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 74f97ce69efe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f050afb
          Default Java 1.8.0_121
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvninstall-root.txt
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvnsite-root.txt
          mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvneclipse-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-distcp hadoop-tools/hadoop-kafka hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 40s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 322 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 15m 11s trunk passed +1 compile 17m 14s trunk passed +1 checkstyle 7m 54s trunk passed +1 mvnsite 10m 25s trunk passed +1 mvneclipse 1m 43s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 25m 54s trunk passed +1 javadoc 4m 32s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch -1 mvninstall 13m 59s root in the patch failed. -1 compile 0m 29s root in the patch failed. -1 javac 0m 29s root in the patch failed. -0 checkstyle 8m 6s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263) -1 mvnsite 0m 24s root in the patch failed. -1 mvneclipse 1m 35s root in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 38s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests +1 findbugs 30m 11s the patch passed -1 javadoc 2m 51s root in the patch failed. -1 unit 7m 12s root in the patch failed. -1 asflicense 0m 38s The patch generated 2 ASF License warnings. 212m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14284 GITHUB PR https://github.com/apache/hadoop/pull/210 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 74f97ce69efe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f050afb Default Java 1.8.0_121 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvnsite-root.txt mvneclipse https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-mvneclipse-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-minicluster hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-distcp hadoop-tools/hadoop-kafka hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12083/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Thanks Akira for trying the patch.

          The failure occurs because both the two module has shaded guava. Would you remove shaded guava from hadoop-client-runtime module?

          Thanks for the suggestion. I could fix it by using shaded Guava in hadoop-shaded-thirdparty, but I rechecked dependencies again before doing so. After rechecking the dependency under hadoop-client-modules, I found that hadoop-client-modules shouldn't use hadoop-shaded-thirdparty. It's because hadoop-shaded-thirdparty exposes Guava's partial classes to use Apache Curator but Apache Curator is only necessary in server side. In hadoop-client-modules, we can shade Guava separately without using hadoop-shaded-thirdparty. I will upload v10 patch with above approach.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Thanks Akira for trying the patch. The failure occurs because both the two module has shaded guava. Would you remove shaded guava from hadoop-client-runtime module? Thanks for the suggestion. I could fix it by using shaded Guava in hadoop-shaded-thirdparty, but I rechecked dependencies again before doing so. After rechecking the dependency under hadoop-client-modules, I found that hadoop-client-modules shouldn't use hadoop-shaded-thirdparty. It's because hadoop-shaded-thirdparty exposes Guava's partial classes to use Apache Curator but Apache Curator is only necessary in server side. In hadoop-client-modules, we can shade Guava separately without using hadoop-shaded-thirdparty. I will upload v10 patch with above approach.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 35s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 322 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 13m 16s trunk passed
          +1 compile 15m 42s trunk passed
          +1 checkstyle 8m 24s trunk passed
          +1 mvnsite 9m 25s trunk passed
          +1 mvneclipse 1m 38s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 24m 24s trunk passed
          +1 javadoc 4m 24s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 39m 43s the patch passed
          -1 compile 0m 27s root in the patch failed.
          -1 javac 0m 27s root in the patch failed.
          -0 checkstyle 8m 30s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263)
          -1 mvnsite 1m 24s root in the patch failed.
          +1 mvneclipse 1m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 39s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 33m 54s the patch passed
          -1 javadoc 3m 24s root in the patch failed.
          -1 unit 8m 18s root in the patch failed.
          -1 asflicense 0m 39s The patch generated 2 ASF License warnings.
          213m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-14284
          GITHUB PR https://github.com/apache/hadoop/pull/210
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 3614463ddda4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f48e968
          Default Java 1.8.0_121
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-mvnsite-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 35s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 322 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 13m 16s trunk passed +1 compile 15m 42s trunk passed +1 checkstyle 8m 24s trunk passed +1 mvnsite 9m 25s trunk passed +1 mvneclipse 1m 38s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 24m 24s trunk passed +1 javadoc 4m 24s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 39m 43s the patch passed -1 compile 0m 27s root in the patch failed. -1 javac 0m 27s root in the patch failed. -0 checkstyle 8m 30s root: The patch generated 150 new + 25248 unchanged - 15 fixed = 25398 total (was 25263) -1 mvnsite 1m 24s root in the patch failed. +1 mvneclipse 1m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 39s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 33m 54s the patch passed -1 javadoc 3m 24s root in the patch failed. -1 unit 8m 18s root in the patch failed. -1 asflicense 0m 39s The patch generated 2 ASF License warnings. 213m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14284 GITHUB PR https://github.com/apache/hadoop/pull/210 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 3614463ddda4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f48e968 Default Java 1.8.0_121 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-mvnsite-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12088/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Attaching v10 patch. Summary:

          • (from v7) Creating hadoop-shaded-thirdparty which includes shaded Guava. However, it doesn't include com.google.common.base.Function, Predicate, and TypeToken. It's because the classes are required to use Apache Curator: https://cwiki.apache.org/confluence/display/CURATOR/TN13
          • (from v7) Using Guava in hadoop-shaded-thirdparty(org.apache.hadoop.com.google.common) in server-side code instead of non-shaded Guava directly. This change subsumes what HADOOP-14238 is addressing.
          • (from v10) Shading Guava inside hadoop-client-modules to shade all Guava unlike hadoop-shaded-thirdparty.
          Show
          ozawa Tsuyoshi Ozawa added a comment - Attaching v10 patch. Summary: (from v7) Creating hadoop-shaded-thirdparty which includes shaded Guava. However, it doesn't include com.google.common.base.Function, Predicate, and TypeToken. It's because the classes are required to use Apache Curator: https://cwiki.apache.org/confluence/display/CURATOR/TN13 (from v7) Using Guava in hadoop-shaded-thirdparty(org.apache.hadoop.com.google.common) in server-side code instead of non-shaded Guava directly. This change subsumes what HADOOP-14238 is addressing. (from v10) Shading Guava inside hadoop-client-modules to shade all Guava unlike hadoop-shaded-thirdparty.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 46s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 322 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 13m 50s trunk passed
          +1 compile 17m 29s trunk passed
          +1 checkstyle 6m 39s trunk passed
          +1 mvnsite 9m 41s trunk passed
          +1 mvneclipse 1m 41s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 25m 57s trunk passed
          +1 javadoc 4m 43s trunk passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 41m 30s the patch passed
          -1 compile 0m 26s root in the patch failed.
          -1 javac 0m 26s root in the patch failed.
          -0 checkstyle 8m 35s root: The patch generated 150 new + 25246 unchanged - 15 fixed = 25396 total (was 25261)
          -1 mvnsite 1m 27s root in the patch failed.
          +1 mvneclipse 1m 41s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 40s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          +1 findbugs 30m 48s the patch passed
          -1 javadoc 2m 51s root in the patch failed.
          -1 unit 7m 36s root in the patch failed.
          -1 asflicense 0m 39s The patch generated 2 ASF License warnings.
          214m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue HADOOP-14284
          GITHUB PR https://github.com/apache/hadoop/pull/210
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 7b688b7fdd42 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0cab572
          Default Java 1.8.0_121
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-mvnsite-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 46s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 322 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 13m 50s trunk passed +1 compile 17m 29s trunk passed +1 checkstyle 6m 39s trunk passed +1 mvnsite 9m 41s trunk passed +1 mvneclipse 1m 41s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 25m 57s trunk passed +1 javadoc 4m 43s trunk passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 41m 30s the patch passed -1 compile 0m 26s root in the patch failed. -1 javac 0m 26s root in the patch failed. -0 checkstyle 8m 35s root: The patch generated 150 new + 25246 unchanged - 15 fixed = 25396 total (was 25261) -1 mvnsite 1m 27s root in the patch failed. +1 mvneclipse 1m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 40s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project +1 findbugs 30m 48s the patch passed -1 javadoc 2m 51s root in the patch failed. -1 unit 7m 36s root in the patch failed. -1 asflicense 0m 39s The patch generated 2 ASF License warnings. 214m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14284 GITHUB PR https://github.com/apache/hadoop/pull/210 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 7b688b7fdd42 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-mvnsite-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-shaded-thirdparty hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-common-project/hadoop-kms hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-distcp hadoop-tools/hadoop-archives hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-aws hadoop-tools/hadoop-kafka hadoop-tools/hadoop-azure hadoop-client-modules/hadoop-client hadoop-tools/hadoop-sls hadoop-tools/hadoop-azure-datalake hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12096/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Compilation failure is caused by not refering maven local repository - how can we install newly added components on test machines? Does anyone know how to do it?

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-auth: Compilation failure: Compilation failure:
          [ERROR] /testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61] error: package org.apache.hadoop.shaded.com.google.common.annotations does not exist

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Compilation failure is caused by not refering maven local repository - how can we install newly added components on test machines? Does anyone know how to do it? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-auth: Compilation failure: Compilation failure: [ERROR] /testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61] error: package org.apache.hadoop.shaded.com.google.common.annotations does not exist
          Hide
          busbey Sean Busbey added a comment -

          This is a problem of build ordering. shading doesn't happen normally until the package phase. But the update to the source means it's needed before then at compile on the dependent modules. Maven doesn't have a notion of this, aside from either a) manually rearranging when shading happens (which will break some assumptions in maven) or b) being different projects rather than just different modules.

          HBase gets around this painfully; we're considering just moving to its own repo and release cycle. maybe Stack can explain it succinctly.

          Show
          busbey Sean Busbey added a comment - This is a problem of build ordering. shading doesn't happen normally until the package phase. But the update to the source means it's needed before then at compile on the dependent modules. Maven doesn't have a notion of this, aside from either a) manually rearranging when shading happens (which will break some assumptions in maven) or b) being different projects rather than just different modules. HBase gets around this painfully; we're considering just moving to its own repo and release cycle. maybe Stack can explain it succinctly.
          Hide
          stack stack added a comment -

          HBase cheats. It has an unofficial pre-build step, the product of which gets checked in so it is available build time (pre-build does stuff like generate class files from protos, shade and patch protobufs so we can run with pb3.....Changes requiring rerun of pre-build are rare). This is messy. We are discussing formalizing pre-build by starting an ancillary project run by the HBase PMC. We'd freight this hbase-3rdparty w/ all of our unofficial pre-build malarky. We also have the 'guava-problem' (and the netty-problem, etc.) and need a soln. Current intent is that hbase-3rdparty includes shaded versions of critical libs (guava, netty, protobuf). Mainline hbase then just includes the hbase-3rdparty artifact.... This is a WIP.

          That referenced Curator TN is an interesting read. Curator made an unfortunate mistake (been there). Propagating their incomplete fix here is unfortunate (Can we depend on a Curator that has complete 'fix'' in hadoop3 or just kick out Curator?).

          Shading Guava inside hadoop-client-modules to shade all Guava unlike hadoop-shaded-thirdparty.

          Does this mean we'd have guava in hadoop-client-modules and in hadoop-shaded-thirdparty? What you thinking Tsuyoshi Ozawa? Thanks.

          Show
          stack stack added a comment - HBase cheats. It has an unofficial pre-build step, the product of which gets checked in so it is available build time (pre-build does stuff like generate class files from protos, shade and patch protobufs so we can run with pb3.....Changes requiring rerun of pre-build are rare). This is messy. We are discussing formalizing pre-build by starting an ancillary project run by the HBase PMC. We'd freight this hbase-3rdparty w/ all of our unofficial pre-build malarky. We also have the 'guava-problem' (and the netty-problem, etc.) and need a soln. Current intent is that hbase-3rdparty includes shaded versions of critical libs (guava, netty, protobuf). Mainline hbase then just includes the hbase-3rdparty artifact.... This is a WIP. That referenced Curator TN is an interesting read. Curator made an unfortunate mistake (been there). Propagating their incomplete fix here is unfortunate (Can we depend on a Curator that has complete 'fix'' in hadoop3 or just kick out Curator?). Shading Guava inside hadoop-client-modules to shade all Guava unlike hadoop-shaded-thirdparty. Does this mean we'd have guava in hadoop-client-modules and in hadoop-shaded-thirdparty? What you thinking Tsuyoshi Ozawa ? Thanks.
          Hide
          andrew.wang Andrew Wang added a comment -

          Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying.

          This patch is also a useful reminder of how much cruft from Guava we're carrying which could potentially be replaced by built-ins from Java 8. Might be worth doing this to help shrink the patch.

          Show
          andrew.wang Andrew Wang added a comment - Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying. This patch is also a useful reminder of how much cruft from Guava we're carrying which could potentially be replaced by built-ins from Java 8. Might be worth doing this to help shrink the patch.
          Hide
          busbey Sean Busbey added a comment -

          > Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying.

          Couldn't we do this shading ourselves? Or worst case, shade our curator use as well?

          Show
          busbey Sean Busbey added a comment - > Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying. Couldn't we do this shading ourselves? Or worst case, shade our curator use as well?
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Does this mean we'd have guava in hadoop-client-modules and in hadoop-shaded-thirdparty? What you thinking Tsuyoshi Ozawa? Thanks.

          Yes, it does. I choosed the way as a workaround because I think hadoop-client-modules shouldn't expose unshaded Guava to end-users of hadoop. Ideally, I would like to enclose shade inside hadoop-shaded-thirdparty completely to avoid its complexity.

          I think using Curator itself is a good choice for stable ZKRMStateStore. There are lots things to use ZK correctly as you know, including connection handling, error handling, etc. Curator does these works instead of us.

          > Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying.
          Couldn't we do this shading ourselves? Or worst case, shade our curator use as well?

          I think we can. For now, I created branch on my github as a trial.

          https://github.com/oza/curator/tree/apache-curator-2.12.0-hadoop

          Its diff from original version is small.
          https://github.com/oza/curator/commit/7bacc5f79cebaec870f30a85f908fadf3edc95ca

          The problem is where to manage the code of forked version and where to push jar file. Under hadoop project?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Does this mean we'd have guava in hadoop-client-modules and in hadoop-shaded-thirdparty? What you thinking Tsuyoshi Ozawa? Thanks. Yes, it does. I choosed the way as a workaround because I think hadoop-client-modules shouldn't expose unshaded Guava to end-users of hadoop. Ideally, I would like to enclose shade inside hadoop-shaded-thirdparty completely to avoid its complexity. I think using Curator itself is a good choice for stable ZKRMStateStore. There are lots things to use ZK correctly as you know, including connection handling, error handling, etc. Curator does these works instead of us. > Could we get Curator to add a new set of APIs that use Curator-shaded Guava objects in their API instead? This would let us fully shade our Guava; the partial shading is unsatisfying. Couldn't we do this shading ourselves? Or worst case, shade our curator use as well? I think we can. For now, I created branch on my github as a trial. https://github.com/oza/curator/tree/apache-curator-2.12.0-hadoop Its diff from original version is small. https://github.com/oza/curator/commit/7bacc5f79cebaec870f30a85f908fadf3edc95ca The problem is where to manage the code of forked version and where to push jar file. Under hadoop project?
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Tsuyoshi Ozawa, looks like your patch shades the API in the Curator build. IIUC Sean Busbey thinks we can consume the official Curator JAR, and rewrite its Guava references with the shade plugin. This would make it easier to maintain in Hadoop.

          What's the minimum viable product, that we can then iterate on? Since Curator only leaks a few Guava classes, I don't consider it a blocker to the new "hadoop-thirdparty" module. Something we can fix before beta1.

          Show
          andrew.wang Andrew Wang added a comment - Hi Tsuyoshi Ozawa , looks like your patch shades the API in the Curator build. IIUC Sean Busbey thinks we can consume the official Curator JAR, and rewrite its Guava references with the shade plugin. This would make it easier to maintain in Hadoop. What's the minimum viable product, that we can then iterate on? Since Curator only leaks a few Guava classes, I don't consider it a blocker to the new "hadoop-thirdparty" module. Something we can fix before beta1.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          Andrew Wang Sean Busbey I see. Let me try in this weekend. Thanks for your pointing.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited Andrew Wang Sean Busbey I see. Let me try in this weekend. Thanks for your pointing.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          80% work to shade Curator and Guava was done. I'm going to post new patch after passing tests locally.

          Show
          ozawa Tsuyoshi Ozawa added a comment - 80% work to shade Curator and Guava was done. I'm going to post new patch after passing tests locally.
          Hide
          djp Junping Du added a comment -

          Sorry for coming late on this. Thanks Tsuyoshi Ozawa for actively working here. It sounds like a huge work.
          A couple of questions: can we just shade some client jars instead of everywhere? Also, can we have some way to skip shade to accelerate maven build process? - current build on trunk is too slow due to shade.

          Show
          djp Junping Du added a comment - Sorry for coming late on this. Thanks Tsuyoshi Ozawa for actively working here. It sounds like a huge work. A couple of questions: can we just shade some client jars instead of everywhere? Also, can we have some way to skip shade to accelerate maven build process? - current build on trunk is too slow due to shade.
          Hide
          busbey Sean Busbey added a comment -

          There already are shaded client jars. There's also a jira for making a shaded jar option for the yarn container (it hasn't moved because of my schedule having a conflict).

          I thought there was a profile to skip shading that got added soon after the shaded client work landed.

          I'm pretty sure unit tests still dominate the build time.

          Show
          busbey Sean Busbey added a comment - There already are shaded client jars. There's also a jira for making a shaded jar option for the yarn container (it hasn't moved because of my schedule having a conflict). I thought there was a profile to skip shading that got added soon after the shaded client work landed. I'm pretty sure unit tests still dominate the build time.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Also, can we have some way to skip shade to accelerate maven build process?

          -DskipShade is the option.

          Show
          ajisakaa Akira Ajisaka added a comment - Also, can we have some way to skip shade to accelerate maven build process? -DskipShade is the option.
          Hide
          aw Allen Wittenauer added a comment -

          I'm pretty sure unit tests still dominate the build time.

          After that, I wouldn't be surprised if it's YARN. It's gotten fat. It is likely 2x-3x what it used to be a few years ago. (esp with TL2 and all of the node.js bits) On disk, all the gains we made by de-duping the dependency jars in trunk (HADOOP-10115) have pretty much been lost due to YARN's growth.

          Show
          aw Allen Wittenauer added a comment - I'm pretty sure unit tests still dominate the build time. After that, I wouldn't be surprised if it's YARN. It's gotten fat . It is likely 2x-3x what it used to be a few years ago. (esp with TL2 and all of the node.js bits) On disk, all the gains we made by de-duping the dependency jars in trunk ( HADOOP-10115 ) have pretty much been lost due to YARN's growth.
          Hide
          djp Junping Du added a comment -

          I'm pretty sure unit tests still dominate the build time.

          I mean skip the unit test.

          After that, I wouldn't be surprised if it's YARN.

          From my local report for mvn install, no evidence to show YARN have any relationship with build slowness. All shading related components are dominating the build speed:

          [INFO] Reactor Summary:
          [INFO]
          [INFO] Apache Hadoop Main ................................. SUCCESS [  1.324 s]
          [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  2.295 s]
          [INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.542 s]
          [INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.355 s]
          [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  0.111 s]
          [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.150 s]
          [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.337 s]
          [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  0.662 s]
          [INFO] Apache Hadoop Auth ................................. SUCCESS [  4.585 s]
          [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.044 s]
          [INFO] Apache Hadoop Common ............................... SUCCESS [ 27.720 s]
          [INFO] Apache Hadoop NFS .................................. SUCCESS [  2.067 s]
          [INFO] Apache Hadoop KMS .................................. SUCCESS [  3.101 s]
          [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.075 s]
          [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 21.961 s]
          [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 27.508 s]
          [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  0.475 s]
          [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.516 s]
          [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  1.081 s]
          [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.049 s]
          [INFO] Apache Hadoop YARN ................................. SUCCESS [  0.052 s]
          [INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.417 s]
          [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 11.146 s]
          [INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.064 s]
          [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  3.441 s]
          [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  5.758 s]
          [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.541 s]
          [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  2.741 s]
          [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.264 s]
          [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.351 s]
          [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  2.159 s]
          [INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.087 s]
          [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.686 s]
          [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.140 s]
          [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  3.577 s]
          [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  4.854 s]
          [INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.083 s]
          [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.650 s]
          [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.062 s]
          [INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.085 s]
          [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  2.026 s]
          [INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.103 s]
          [INFO] Apache Hadoop YARN Project ......................... SUCCESS [  2.950 s]
          [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.269 s]
          [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [  7.138 s]
          [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  4.366 s]
          [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.704 s]
          [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.262 s]
          [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.684 s]
          [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  6.478 s]
          [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.172 s]
          [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  2.150 s]
          [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.053 s]
          [INFO] Apache Hadoop MapReduce ............................ SUCCESS [  0.290 s]
          [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  1.686 s]
          [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.631 s]
          [INFO] Apache Hadoop Archives ............................. SUCCESS [  0.953 s]
          [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.250 s]
          [INFO] Apache Hadoop Rumen ................................ SUCCESS [  1.696 s]
          [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  1.670 s]
          [INFO] Apache Hadoop Data Join ............................ SUCCESS [  0.813 s]
          [INFO] Apache Hadoop Extras ............................... SUCCESS [  0.935 s]
          [INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.071 s]
          [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  1.317 s]
          [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  2.900 s]
          [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  1.079 s]
          [INFO] Apache Hadoop Azure support ........................ SUCCESS [  2.465 s]
          [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.354 s]
          [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  3.009 s]
          [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.571 s]
          [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.093 s]
          [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.597 s]
          [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  1.670 s]
          [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.060 s]
          [INFO] Apache Hadoop Client API ........................... SUCCESS [02:29 min]
          [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:11 min]
          [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.275 s]
          [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:59 min]
          [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.190 s]
          [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.224 s]
          [INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.253 s]
          [INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.053 s]
          [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.970 s]
          [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.039 s]
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 10:02 min
          [INFO] Finished at: 2017-05-01T17:09:55-07:00
          [INFO] Final Memory: 143M/500M
          [INFO] ------------------------------------------------------------------------
          

          Comparing with branch-2, build time for "Apache Hadoop Client" are increasing from 2+ s to several minutes (for API, Runtime, etc.). Also, build of Minicluster are changing from 1.3 s to 2 minutes, etc. Still negligible change for build process?

          Show
          djp Junping Du added a comment - I'm pretty sure unit tests still dominate the build time. I mean skip the unit test. After that, I wouldn't be surprised if it's YARN. From my local report for mvn install, no evidence to show YARN have any relationship with build slowness. All shading related components are dominating the build speed: [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 1.324 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.295 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.542 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.355 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.111 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.150 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.337 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.662 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.585 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.044 s] [INFO] Apache Hadoop Common ............................... SUCCESS [ 27.720 s] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.067 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.101 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.075 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 21.961 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 27.508 s] [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 0.475 s] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.516 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.081 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.049 s] [INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.052 s] [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.417 s] [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 11.146 s] [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.064 s] [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 3.441 s] [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 5.758 s] [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.541 s] [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 2.741 s] [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.264 s] [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.351 s] [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 2.159 s] [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.087 s] [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.686 s] [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.140 s] [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 3.577 s] [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 4.854 s] [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.083 s] [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.650 s] [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.062 s] [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.085 s] [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.026 s] [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.103 s] [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 2.950 s] [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.269 s] [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 7.138 s] [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 4.366 s] [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.704 s] [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.262 s] [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.684 s] [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 6.478 s] [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.172 s] [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 2.150 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.053 s] [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 0.290 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 1.686 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.631 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 0.953 s] [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.250 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 1.696 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.670 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 0.813 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 0.935 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.071 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.317 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 2.900 s] [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.079 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 2.465 s] [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.354 s] [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 3.009 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 2.571 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.093 s] [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.597 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 1.670 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.060 s] [INFO] Apache Hadoop Client API ........................... SUCCESS [02:29 min] [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:11 min] [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.275 s] [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:59 min] [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.190 s] [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.224 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 0.253 s] [INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.053 s] [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.970 s] [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.039 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10:02 min [INFO] Finished at: 2017-05-01T17:09:55-07:00 [INFO] Final Memory: 143M/500M [INFO] ------------------------------------------------------------------------ Comparing with branch-2, build time for "Apache Hadoop Client" are increasing from 2+ s to several minutes (for API, Runtime, etc.). Also, build of Minicluster are changing from 1.3 s to 2 minutes, etc. Still negligible change for build process?
          Hide
          andrew.wang Andrew Wang added a comment -

          To clarify a little, I do not want skipShade to apply to the third-party shading being done here. We support both shaded and unshaded client JARs, and skipShade is useful to skip creating the shaded client jars. Here though, we want these third-party JARs shaded everywhere, and the shaded JARs will be the only supported artifacts.

          Show
          andrew.wang Andrew Wang added a comment - To clarify a little, I do not want skipShade to apply to the third-party shading being done here. We support both shaded and unshaded client JARs, and skipShade is useful to skip creating the shaded client jars. Here though, we want these third-party JARs shaded everywhere, and the shaded JARs will be the only supported artifacts.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Are we sure this is the best approach? The 1.1MB patch just to shade guava is downright scary. As we keep doing this for other libraries, I'm concerned if our code becomes more brittle (changing imports everywhere) and if the build times explode.

          Are there alternatives? Isn't it better to just shade our final artifacts instead of shading individual libraries' jars? I remember doing something of this sort in the very early stages of YARN.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Are we sure this is the best approach? The 1.1MB patch just to shade guava is downright scary. As we keep doing this for other libraries, I'm concerned if our code becomes more brittle (changing imports everywhere) and if the build times explode. Are there alternatives? Isn't it better to just shade our final artifacts instead of shading individual libraries' jars? I remember doing something of this sort in the very early stages of YARN.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Vinod Kumar Vavilapalli and Junping Du, thanks a lot for your feedback.

          > A couple of questions: can we just shade some client jars instead of everywhere?
          > As we keep doing this for other libraries, I'm concerned if our code becomes more brittle (changing imports everywhere) and if the build times explode.

          We're now doing to shade Guava and Curator in hadoop-shaded-thirdparty, and trying to import it from hadoop-* projects. The build time of Hadoop doesn't get increased so much with this approach because of just referring to hadoop-shaded-thirdparty project from hadoop-* projects. However, I found one problem in this approach: shaded artifacts(shaded Guava and Curator) in hadoop-shaded-thirdparty is NOT in classpath, if I understand correctly. To go with this approach, we need to unzip source code and compile it like HBase does in hbase-protocol-shaded. This can make Hadoop build fragile and the build time of Hadoop can increase as Junping Vinod mentioned.

          https://github.com/apache/hbase/blob/7700a7fac1262934fe538a96b040793c6ff171ce/hbase-protocol-shaded/pom.xml#L321

          Gradle seems to have a feature to do this.

          http://stackoverflow.com/questions/26244936/how-to-include-only-project-and-relocated-classes-when-using-gradle-shadow-plugi

          > Isn't it better to just shade our final artifacts instead of shading individual libraries' jars?

          Do you mean that we prepare new project "hadoop-server-modules" and shading Guava and Curator inside them like hadoop-client-modules? It sounds better approach to me. By adding skipShade option here, we can overcome build time problem. Andrew Wang Sean Busbey Akira Ajisaka What do you think?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Vinod Kumar Vavilapalli and Junping Du , thanks a lot for your feedback. > A couple of questions: can we just shade some client jars instead of everywhere? > As we keep doing this for other libraries, I'm concerned if our code becomes more brittle (changing imports everywhere) and if the build times explode. We're now doing to shade Guava and Curator in hadoop-shaded-thirdparty, and trying to import it from hadoop-* projects. The build time of Hadoop doesn't get increased so much with this approach because of just referring to hadoop-shaded-thirdparty project from hadoop-* projects. However, I found one problem in this approach: shaded artifacts(shaded Guava and Curator) in hadoop-shaded-thirdparty is NOT in classpath, if I understand correctly. To go with this approach, we need to unzip source code and compile it like HBase does in hbase-protocol-shaded. This can make Hadoop build fragile and the build time of Hadoop can increase as Junping Vinod mentioned. https://github.com/apache/hbase/blob/7700a7fac1262934fe538a96b040793c6ff171ce/hbase-protocol-shaded/pom.xml#L321 Gradle seems to have a feature to do this. http://stackoverflow.com/questions/26244936/how-to-include-only-project-and-relocated-classes-when-using-gradle-shadow-plugi > Isn't it better to just shade our final artifacts instead of shading individual libraries' jars? Do you mean that we prepare new project "hadoop-server-modules" and shading Guava and Curator inside them like hadoop-client-modules? It sounds better approach to me. By adding skipShade option here, we can overcome build time problem. Andrew Wang Sean Busbey Akira Ajisaka What do you think?
          Hide
          stack stack added a comment -

          However, I found one problem in this approach: shaded artifacts(shaded Guava and Curator) in hadoop-shaded-thirdparty is NOT in classpath, if I understand correctly.

          Tell us more please. Shading bundles the relocated .class files of guava and curator; they are included in the thirdparty jar... and the thirdparty jar is on the classpath, no?

          Perhaps you are referring to the downsides listed in the comment 'stack added a comment - 06/Apr/17 00:57' over in HADOOP-13363 where IDEs will not be able to find the shaded imports? For this reason hbase-protocol-shaded includes relocated src (it does a build because we patch protobuf3).

          Show
          stack stack added a comment - However, I found one problem in this approach: shaded artifacts(shaded Guava and Curator) in hadoop-shaded-thirdparty is NOT in classpath, if I understand correctly. Tell us more please. Shading bundles the relocated .class files of guava and curator; they are included in the thirdparty jar... and the thirdparty jar is on the classpath, no? Perhaps you are referring to the downsides listed in the comment 'stack added a comment - 06/Apr/17 00:57' over in HADOOP-13363 where IDEs will not be able to find the shaded imports? For this reason hbase-protocol-shaded includes relocated src (it does a build because we patch protobuf3).
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          stack thanks you for your help.

          Tell us more please. Shading bundles the relocated .class files of guava and curator; they are included in the thirdparty jar... and the thirdparty jar is on the classpath, no?

          Yes, they are. This is a list of the contents of hadoop-shaded-thirdparty: https://gist.github.com/oza/62fdea66a55c86eda02d2a8530058153
          The list shows that shading works well. As a result, mvn install -DskipTests succeeds with the patch: it means the thirdparty jar is on the classpath at compile time.

          On the other hand, mvn test of hadoop-auth fails with the patch although I added dependency on hadoop-shaded-thirdparty in compile scope.

          https://github.com/oza/hadoop/blob/HADOOP-14284/hadoop-common-project/hadoop-auth/pom.xml#L46

          the result of mvn test is as follows:

          Running org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator
          Tests run: 20, Failures: 5, Errors: 11, Skipped: 0, Time elapsed: 23.407 sec <<< FAILURE! - in org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator
          testNotAuthenticatedWithMultiAuthHandler[0](org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator) Time elapsed: 1.649 sec <<< ERROR!
          java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/google/common/base/Splitter

          I thought that this might be a problem because the thirdparty jar is not on the classpath at testing time, Please let me know if I'm missing something.

          Show
          ozawa Tsuyoshi Ozawa added a comment - stack thanks you for your help. Tell us more please. Shading bundles the relocated .class files of guava and curator; they are included in the thirdparty jar... and the thirdparty jar is on the classpath, no? Yes, they are. This is a list of the contents of hadoop-shaded-thirdparty: https://gist.github.com/oza/62fdea66a55c86eda02d2a8530058153 The list shows that shading works well. As a result, mvn install -DskipTests succeeds with the patch: it means the thirdparty jar is on the classpath at compile time. On the other hand, mvn test of hadoop-auth fails with the patch although I added dependency on hadoop-shaded-thirdparty in compile scope. https://github.com/oza/hadoop/blob/HADOOP-14284/hadoop-common-project/hadoop-auth/pom.xml#L46 the result of mvn test is as follows: Running org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator Tests run: 20, Failures: 5, Errors: 11, Skipped: 0, Time elapsed: 23.407 sec <<< FAILURE! - in org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator testNotAuthenticatedWithMultiAuthHandler[0](org.apache.hadoop.security.authentication.client.TestKerberosAuthenticator) Time elapsed: 1.649 sec <<< ERROR! java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/google/common/base/Splitter I thought that this might be a problem because the thirdparty jar is not on the classpath at testing time, Please let me know if I'm missing something.
          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          After making shadeTestJar true in hadoop-shaded-thirdparty jar, it works! Attaching patch soon.

          This was wrong. It still had same error message unfortunately.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited After making shadeTestJar true in hadoop-shaded-thirdparty jar, it works! Attaching patch soon. This was wrong. It still had same error message unfortunately.
          Hide
          djp Junping Du added a comment -

          Instead of shade guava everywhere, do we consider the option to remove guava completely in our code base? I like the convenience we gain in dev stage, but really hate the trouble in maintaining/releases given poor compatibility across versions. Shall we just remove it just like what other projects (like ElasticSearch, etc.) did before?

          Show
          djp Junping Du added a comment - Instead of shade guava everywhere, do we consider the option to remove guava completely in our code base? I like the convenience we gain in dev stage, but really hate the trouble in maintaining/releases given poor compatibility across versions. Shall we just remove it just like what other projects (like ElasticSearch, etc.) did before?
          Hide
          busbey Sean Busbey added a comment -

          I thought that this might be a problem because the thirdparty jar is not on the classpath at testing time, Please let me know if I'm missing something.

          This is because the shading relocation doesn't happen until the package phase, which is after the test phase. Maven goes through a given phase for all modules in a reactor before doing the next one. If you look at the integration tests for the shaded client, you can see where the compilation of test classes there are moved to a new cycle after packaging has already shaded everything.

          For this, you'd either have to move the shading for the third party jar to be before the normal compile phase or we'd need it in a different repo with its own build cycle.

          Show
          busbey Sean Busbey added a comment - I thought that this might be a problem because the thirdparty jar is not on the classpath at testing time, Please let me know if I'm missing something. This is because the shading relocation doesn't happen until the package phase, which is after the test phase. Maven goes through a given phase for all modules in a reactor before doing the next one. If you look at the integration tests for the shaded client, you can see where the compilation of test classes there are moved to a new cycle after packaging has already shaded everything. For this, you'd either have to move the shading for the third party jar to be before the normal compile phase or we'd need it in a different repo with its own build cycle.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Junping Du It's another option. IMHO, it sounds more tough work than we expected - e.g. HDFS core including DataNode.java uses CacheBuilder. Could you share the examples that can be caused even with shading? The patch looks large, but it's just a change of importing and pom.xml.

          Sean Busbey thanks a lot! Your advice is correct. I used jarjar-maven-plugin to be before the package phase(I'm using process-classes to pass tests for now) because shade:shade is bound to the package phase. The v12 patch includes the change. PR for review on github is available here.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Junping Du It's another option. IMHO, it sounds more tough work than we expected - e.g. HDFS core including DataNode.java uses CacheBuilder. Could you share the examples that can be caused even with shading? The patch looks large, but it's just a change of importing and pom.xml. Sean Busbey thanks a lot! Your advice is correct. I used jarjar-maven-plugin to be before the package phase( I'm using process-classes to pass tests for now ) because shade:shade is bound to the package phase. The v12 patch includes the change. PR for review on github is available here .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 328 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 30s trunk passed
          +1 compile 16m 18s trunk passed
          +1 checkstyle 8m 50s trunk passed
          +1 mvnsite 9m 28s trunk passed
          +1 mvneclipse 1m 37s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project
          -1 findbugs 0m 27s hadoop-common-project/hadoop-auth in trunk has 1 extant Findbugs warnings.
          -1 findbugs 1m 18s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          -1 findbugs 1m 25s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings.
          -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          -1 findbugs 0m 59s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings.
          -1 findbugs 0m 46s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
          -1 findbugs 0m 32s hadoop-tools/hadoop-azure in trunk has 1 extant Findbugs warnings.
          -1 findbugs 0m 32s hadoop-tools/hadoop-sls in trunk has 1 extant Findbugs warnings.
          +1 javadoc 4m 25s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 42m 51s the patch passed
          +1 compile 15m 13s the patch passed
          -1 javac 15m 13s root generated 15 new + 772 unchanged - 15 fixed = 787 total (was 787)
          -0 checkstyle 9m 57s root: The patch generated 153 new + 25278 unchanged - 15 fixed = 25431 total (was 25293)
          +1 mvnsite 11m 18s the patch passed
          +1 mvneclipse 1m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 47s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
          +1 findbugs 32m 56s the patch passed
          -1 javadoc 2m 53s root in the patch failed.
          -1 unit 125m 39s root in the patch failed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          360m 4s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
            hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
            hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
            hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
            hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
            hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
          Timed out junit tests org.apache.hadoop.hdfs.TestReplication



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14284
          GITHUB PR https://github.com/apache/hadoop/pull/210
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux bcd31966ab68 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d6eed5a
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-auth-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-azure-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-sls-warnings.html
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/patch-javadoc-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/testReport/
          modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-auth-examples hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-distcp hadoop-tools/hadoop-kafka hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 328 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 30s trunk passed +1 compile 16m 18s trunk passed +1 checkstyle 8m 50s trunk passed +1 mvnsite 9m 28s trunk passed +1 mvneclipse 1m 37s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster . hadoop-mapreduce-project -1 findbugs 0m 27s hadoop-common-project/hadoop-auth in trunk has 1 extant Findbugs warnings. -1 findbugs 1m 18s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. -1 findbugs 1m 25s hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. -1 findbugs 1m 43s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. -1 findbugs 0m 59s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant Findbugs warnings. -1 findbugs 0m 46s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. -1 findbugs 0m 32s hadoop-tools/hadoop-azure in trunk has 1 extant Findbugs warnings. -1 findbugs 0m 32s hadoop-tools/hadoop-sls in trunk has 1 extant Findbugs warnings. +1 javadoc 4m 25s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 42m 51s the patch passed +1 compile 15m 13s the patch passed -1 javac 15m 13s root generated 15 new + 772 unchanged - 15 fixed = 787 total (was 787) -0 checkstyle 9m 57s root: The patch generated 153 new + 25278 unchanged - 15 fixed = 25431 total (was 25293) +1 mvnsite 11m 18s the patch passed +1 mvneclipse 1m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 47s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests +1 findbugs 32m 56s the patch passed -1 javadoc 2m 53s root in the patch failed. -1 unit 125m 39s root in the patch failed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 360m 4s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity   hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage   hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities Timed out junit tests org.apache.hadoop.hdfs.TestReplication Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14284 GITHUB PR https://github.com/apache/hadoop/pull/210 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux bcd31966ab68 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d6eed5a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-auth-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-azure-warnings.html findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-sls-warnings.html javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/patch-javadoc-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/testReport/ modules C: hadoop-project hadoop-shaded-thirdparty . hadoop-client-modules/hadoop-client hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-runtime hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-auth-examples hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-hdfs-project/hadoop-hdfs-nfs hadoop-mapreduce-project hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-tools/hadoop-archive-logs hadoop-tools/hadoop-archives hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-azure-datalake hadoop-tools/hadoop-distcp hadoop-tools/hadoop-kafka hadoop-tools/hadoop-sls hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12257/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Folks, do we think we can get this turned around by end-of-week for alpha3? It might be worth holding the release by a little if we think it's close.

          Show
          andrew.wang Andrew Wang added a comment - Folks, do we think we can get this turned around by end-of-week for alpha3? It might be worth holding the release by a little if we think it's close.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Andrew Wang Once HADOOP-14386 is merged, I think we can finish this jira soon. The test failures we're facing with the latest patch is caused by updating Guava in timelineserver test module to the newer version than 0.14, which is the version removing LimitInputStream. After HADOOP-14386, the problem will be solved at the implementation level.
          More important point is that whether we will reach consensus or not.

          Junping Du Vinod Kumar Vavilapalli could you share your opinion based on the information we shared after your comments? The looks large patch, however, it only changes the code of import statement and pom.xml. diff size is as follows as a reference:

          $ git diff --shortstat d6eed5acca222a281401c1d524824f0460ea7826
          1241 files changed, 2946 insertions, 2137 deletions 1241 files changed, 2946 insertions, 2137 deletions

          I think maintaining all of staffs of Guava in Hadoop side is also tough work, so shading Guava looks to me like a good solution. The Guava problem is an ecosystem-wide problem as you know. If we can solve here, it's good for Hadoop ecosystem.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Andrew Wang Once HADOOP-14386 is merged, I think we can finish this jira soon. The test failures we're facing with the latest patch is caused by updating Guava in timelineserver test module to the newer version than 0.14, which is the version removing LimitInputStream. After HADOOP-14386 , the problem will be solved at the implementation level. More important point is that whether we will reach consensus or not. Junping Du Vinod Kumar Vavilapalli could you share your opinion based on the information we shared after your comments? The looks large patch, however, it only changes the code of import statement and pom.xml. diff size is as follows as a reference: $ git diff --shortstat d6eed5acca222a281401c1d524824f0460ea7826 1241 files changed, 2946 insertions , 2137 deletions 1241 files changed, 2946 insertions , 2137 deletions I think maintaining all of staffs of Guava in Hadoop side is also tough work, so shading Guava looks to me like a good solution. The Guava problem is an ecosystem-wide problem as you know. If we can solve here, it's good for Hadoop ecosystem.
          Hide
          djp Junping Du added a comment -

          Tsuyoshi Ozawa, I appreciate your work and effort here. However, I still doubt if we are on the right track here with right solution. IMO, shade dependencies on client jars is an acceptable solution but effort like this to shade particular poor backward compatibility dependency everywhere seems unacceptable in build prospective - even worse case is if other dependencies are following this practice, then it would be nightmare for future maintaining of hadoop.
          Considering Hadoop ecosystem has large number of downstream projects, we should keep core hadoop as stable as possible - even in a new major release. For poor backward compatibility third-party jar, like Guava, I think we should either get rid of it or keep on a particular version and never change . My proposal above to get rid of Guava seems too aggressive and unrealistic in short term. However, the other approach - to stay at a version with other downstream projects unless must be changed (and should change with downstream projects together) - seems the best strategy here. I didn't see any necessary for upgrading guava and HADOOP-10101 failed to convince me. In our test internally, it cause Hadoop 3 failed to work with almost every downstream projects, like: Spark, HBase, Tez, etc. I think we should revert HADOOP-10101 immediately before it is too late.

          Show
          djp Junping Du added a comment - Tsuyoshi Ozawa , I appreciate your work and effort here. However, I still doubt if we are on the right track here with right solution. IMO, shade dependencies on client jars is an acceptable solution but effort like this to shade particular poor backward compatibility dependency everywhere seems unacceptable in build prospective - even worse case is if other dependencies are following this practice, then it would be nightmare for future maintaining of hadoop. Considering Hadoop ecosystem has large number of downstream projects, we should keep core hadoop as stable as possible - even in a new major release. For poor backward compatibility third-party jar, like Guava, I think we should either get rid of it or keep on a particular version and never change . My proposal above to get rid of Guava seems too aggressive and unrealistic in short term. However, the other approach - to stay at a version with other downstream projects unless must be changed (and should change with downstream projects together) - seems the best strategy here. I didn't see any necessary for upgrading guava and HADOOP-10101 failed to convince me. In our test internally, it cause Hadoop 3 failed to work with almost every downstream projects, like: Spark, HBase, Tez, etc. I think we should revert HADOOP-10101 immediately before it is too late.
          Hide
          aw Allen Wittenauer added a comment -

          keep on a particular version and never change

          This isn't really a viable option long-term. Ignoring problems like security issues, guava has already been broken by changes in the JVM in the past. There's no reason to believe it won't happen again. The decisions made in Hadoop 3.x should be viewed as an investment in the long term.

          Show
          aw Allen Wittenauer added a comment - keep on a particular version and never change This isn't really a viable option long-term. Ignoring problems like security issues, guava has already been broken by changes in the JVM in the past. There's no reason to believe it won't happen again. The decisions made in Hadoop 3.x should be viewed as an investment in the long term.
          Hide
          stack stack added a comment -

          ...get rid of it....

          This doesn't seem right either mighty Junping Du. Guava is a high quality lib. It is well tested with a particular attention paid to perf. I'd think we'd want to double-down on libs of this type rather than move to purge it. Would netty/protobuf/etc. be next in line sir?

          Tsuyoshi Ozawa Have you brought up the project in an IDE with your patch applied? My expectation is that there will reams of complaint that the relocated guava can't be found....

          Show
          stack stack added a comment - ...get rid of it.... This doesn't seem right either mighty Junping Du . Guava is a high quality lib. It is well tested with a particular attention paid to perf. I'd think we'd want to double-down on libs of this type rather than move to purge it. Would netty/protobuf/etc. be next in line sir? Tsuyoshi Ozawa Have you brought up the project in an IDE with your patch applied? My expectation is that there will reams of complaint that the relocated guava can't be found....
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Junping Du As I mentioned on HADOOP-14386, I'm okay to downgrade version of Guava to 0.11.2. On the other hand, I'm still thinking that shading itself is very useful.

          However, the other approach - to stay at a version with other downstream projects unless must be changed (and should change with downstream projects together) - seems the best strategy here. I didn't see any necessary for upgrading guava and HADOOP-10101 failed to convince me.

          First of all, the opinions of users on HADOOP-10101 looks to me like positive against shading Guava. Secondly, as stack mentioned, the quality of Guava itself is nice. Guava also announces deprecated methods in the future on ahead. For example. https://google.github.io/guava/releases/14.0/api/docs/com/google/common/io/LimitInputStream.html
          If we shade and relocate Guava inside Hadoop, we can catch up these changes easily because the dependency doesn't leak to user-level. It's not a problem of Guava, IIUC - it's a problem that we failed to decouple with user-level library and server-side library. There is a possibility that the reams of complaint come from IDE problem. Hence, I will post a next patch with IDE support after HADOOP-14386.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Junping Du As I mentioned on HADOOP-14386 , I'm okay to downgrade version of Guava to 0.11.2. On the other hand, I'm still thinking that shading itself is very useful. However, the other approach - to stay at a version with other downstream projects unless must be changed (and should change with downstream projects together) - seems the best strategy here. I didn't see any necessary for upgrading guava and HADOOP-10101 failed to convince me. First of all, the opinions of users on HADOOP-10101 looks to me like positive against shading Guava. Secondly, as stack mentioned, the quality of Guava itself is nice. Guava also announces deprecated methods in the future on ahead. For example. https://google.github.io/guava/releases/14.0/api/docs/com/google/common/io/LimitInputStream.html If we shade and relocate Guava inside Hadoop, we can catch up these changes easily because the dependency doesn't leak to user-level. It's not a problem of Guava, IIUC - it's a problem that we failed to decouple with user-level library and server-side library. There is a possibility that the reams of complaint come from IDE problem. Hence, I will post a next patch with IDE support after HADOOP-14386 .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Junping Du Vinod Kumar Vavilapalli could you share your opinion based on the information we shared after your comments? The looks large patch, however, it only changes the code of import statement and pom.xml. diff size is as follows as a reference:

          Tsuyoshi Ozawa, I think there is one thing we should definitely do. YARN and mapreduce have always had separate client libraries. And the expectation of users has always been to not depend on the server jars. So you should definitely skip YARN server side modules and mapreduce non-client modules from shading.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Junping Du Vinod Kumar Vavilapalli could you share your opinion based on the information we shared after your comments? The looks large patch, however, it only changes the code of import statement and pom.xml. diff size is as follows as a reference: Tsuyoshi Ozawa , I think there is one thing we should definitely do. YARN and mapreduce have always had separate client libraries. And the expectation of users has always been to not depend on the server jars. So you should definitely skip YARN server side modules and mapreduce non-client modules from shading.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Isn't it better to just shade our final artifacts instead of shading individual libraries' jars?

          Do you mean that we prepare new project "hadoop-server-modules" and shading Guava and Curator inside them like hadoop-client-modules? It sounds better approach to me. By adding skipShade option here, we can overcome build time problem. Andrew Wang Sean Busbey Akira Ajisaka What do you think?

          Tsuyoshi Ozawa, I meant that we shade the leaf modules. For e.g. we shade hadoop-yarn-client module (instead of guava) completely so that the downstream users of hadoop-yarn-client only see one big fat jar with all the dependencies shaded. This way guava, log4j that hadoop-yarn-client depends on are completely invisible to users. The downside of this is that (a) the fat jar becomes, ahem, fat (b) some folks may want to control the versions explicitly. (a) is unavoidable. For (b), we could ship both slim and fat jars.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Isn't it better to just shade our final artifacts instead of shading individual libraries' jars? Do you mean that we prepare new project "hadoop-server-modules" and shading Guava and Curator inside them like hadoop-client-modules? It sounds better approach to me. By adding skipShade option here, we can overcome build time problem. Andrew Wang Sean Busbey Akira Ajisaka What do you think? Tsuyoshi Ozawa , I meant that we shade the leaf modules. For e.g. we shade hadoop-yarn-client module (instead of guava) completely so that the downstream users of hadoop-yarn-client only see one big fat jar with all the dependencies shaded. This way guava, log4j that hadoop-yarn-client depends on are completely invisible to users. The downside of this is that (a) the fat jar becomes, ahem, fat (b) some folks may want to control the versions explicitly. (a) is unavoidable. For (b), we could ship both slim and fat jars.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          One thing Tsuyoshi Ozawa calls out: once we shade we can be more aggressive in using guava bits, as we don't have to worry about downstream breakage. What we do have to do is be consistent in shadeage , and confident that there's no leakage of guava structures, not just in "public" APIs, but in common override points/subclassings.

          I can see we'll need to do this for all the cloud storage modules too; cnauroth added a hadoop-cloudprojects POM module, but maybe we'll need to do something which shades.

          Show
          stevel@apache.org Steve Loughran added a comment - One thing Tsuyoshi Ozawa calls out: once we shade we can be more aggressive in using guava bits, as we don't have to worry about downstream breakage. What we do have to do is be consistent in shadeage , and confident that there's no leakage of guava structures, not just in "public" APIs, but in common override points/subclassings. I can see we'll need to do this for all the cloud storage modules too; cnauroth added a hadoop-cloudprojects POM module, but maybe we'll need to do something which shades.
          Hide
          busbey Sean Busbey added a comment -

          It sounds like this discussion is sliding towards YARN-6466 to compliment the shaded client artifacts. Is there any part of this that that effort wouldn't cover (presuming our beliefs about API leakage are correct)?

          Show
          busbey Sean Busbey added a comment - It sounds like this discussion is sliding towards YARN-6466 to compliment the shaded client artifacts. Is there any part of this that that effort wouldn't cover (presuming our beliefs about API leakage are correct)?
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Is there any part of this that that effort wouldn't cover (presuming our beliefs about API leakage are correct)?

          We would need a solution for the hadoop-common Jar which depends on Guava, perhaps a hadoop-common-runtime fat jar.

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Is there any part of this that that effort wouldn't cover (presuming our beliefs about API leakage are correct)? We would need a solution for the hadoop-common Jar which depends on Guava, perhaps a hadoop-common-runtime fat jar.
          Hide
          busbey Sean Busbey added a comment -

          where would such a hadoop-common jar get used?

          Show
          busbey Sean Busbey added a comment - where would such a hadoop-common jar get used?
          Hide
          wheat9 Haohui Mai added a comment -

          Upgrading guava would help us a lot as many of our internal users (tens of teams) have been using Guava 21. Because many of them are on the Apache ecosystem (Hadoop / Spark / Flink / Hive / etc.), accommodating guava 11 and 21 is definitely not fun.

          I'm more align with Steve Loughran, it looks to me that shading Guava is a step forward instead of a step backward.

          To Junping Du's concerns – I think it can be largely addressed by improving our POMs to automate the process. Thoughts?

          Show
          wheat9 Haohui Mai added a comment - Upgrading guava would help us a lot as many of our internal users (tens of teams) have been using Guava 21. Because many of them are on the Apache ecosystem (Hadoop / Spark / Flink / Hive / etc.), accommodating guava 11 and 21 is definitely not fun. I'm more align with Steve Loughran , it looks to me that shading Guava is a step forward instead of a step backward. To Junping Du 's concerns – I think it can be largely addressed by improving our POMs to automate the process. Thoughts?
          Hide
          andrew.wang Andrew Wang added a comment -

          I'd like to try and move this forward, the other 3.0.0-alpha4 blockers are making good progress. We need this one to unblock the PB upgrade JIRA as well.

          If I understand Vinod correctly, he's alright with shading approach if we don't shade the YARN server artifacts and MR non-client libs. That makes sense to me.

          Unfortunately for HDFS, there are a bunch of downstreams incorrectly including our server artifacts, so for HDFS, I think we need to shade those too.

          Tsuyoshi Ozawa are we in a state where we can move forward on this JIRA?

          Show
          andrew.wang Andrew Wang added a comment - I'd like to try and move this forward, the other 3.0.0-alpha4 blockers are making good progress. We need this one to unblock the PB upgrade JIRA as well. If I understand Vinod correctly, he's alright with shading approach if we don't shade the YARN server artifacts and MR non-client libs. That makes sense to me. Unfortunately for HDFS, there are a bunch of downstreams incorrectly including our server artifacts, so for HDFS, I think we need to shade those too. Tsuyoshi Ozawa are we in a state where we can move forward on this JIRA?
          Hide
          kasha Karthik Kambatla added a comment -

          I agree with Vinod Kumar Vavilapalli on shading only the YARN/MR client modules.

          For YARN, that is yarn-common, yarn-client, and yarn-api modules. For MR, that should be mapreduce-client-* modules. We probably don't need to shade hadoop-mapreduce-client-hs and hadoop-mapreduce-client-hs-plugins jars though, as they are for the HistoryServer and have no @Stable APIs.

          In cases where devs extend YARN classes like the SchedulingPolicy in FairScheduler or implement their own scheduler, the dev will be responsible for ensuring they don't use guava or use it with a version that is consistent with what Hadoop uses. I expect these devs to be sophisticated enough to figure this out. That said, we should probably still call out cases like these in the compatibility guide. /cc Daniel Templeton

          Show
          kasha Karthik Kambatla added a comment - I agree with Vinod Kumar Vavilapalli on shading only the YARN/MR client modules. For YARN, that is yarn-common, yarn-client, and yarn-api modules. For MR, that should be mapreduce-client-* modules. We probably don't need to shade hadoop-mapreduce-client-hs and hadoop-mapreduce-client-hs-plugins jars though, as they are for the HistoryServer and have no @Stable APIs. In cases where devs extend YARN classes like the SchedulingPolicy in FairScheduler or implement their own scheduler, the dev will be responsible for ensuring they don't use guava or use it with a version that is consistent with what Hadoop uses. I expect these devs to be sophisticated enough to figure this out. That said, we should probably still call out cases like these in the compatibility guide. /cc Daniel Templeton
          Hide
          busbey Sean Busbey added a comment -

          I don't understand why we'd shade just the client modules for YARN. If a user wants Guava out of the classpath for the client side, why not just use the shaded client? In either of those approaches folks are still out of luck on the server side. I thought that was the point of this jira?

          Unfortunately for HDFS, there are a bunch of downstreams incorrectly including our server artifacts, so for HDFS, I think we need to shade those too.

          This seems like the wrong approach IMHO. There's no incentive for folks to ever start respecting the public/not public interface we put up. If we're only concerned about client side impact, then we should just upgrade guava and close this jira. That way if folks run into a problem with the guava uprade, we just tell them to use the correct client jars instead of server jars. we can even put it in the release note for the guava upgrade.

          Show
          busbey Sean Busbey added a comment - I don't understand why we'd shade just the client modules for YARN. If a user wants Guava out of the classpath for the client side, why not just use the shaded client? In either of those approaches folks are still out of luck on the server side. I thought that was the point of this jira? Unfortunately for HDFS, there are a bunch of downstreams incorrectly including our server artifacts, so for HDFS, I think we need to shade those too. This seems like the wrong approach IMHO. There's no incentive for folks to ever start respecting the public/not public interface we put up. If we're only concerned about client side impact, then we should just upgrade guava and close this jira. That way if folks run into a problem with the guava uprade, we just tell them to use the correct client jars instead of server jars. we can even put it in the release note for the guava upgrade.
          Hide
          djp Junping Du added a comment -

          If a user wants Guava out of the classpath for the client side, why not just use the shaded client?

          I think we are working towards this direction. My understanding is YARN/MR client haven't finish shade practice. Isn't it?

          In either of those approaches folks are still out of luck on the server side. I thought that was the point of this jira?

          There shouldn't be too much concern on server side as application will have separated classloader so different version of guava should get loaded for different places. IMO, this JIRA approach is hardly being accepted not only because its own complexity but also open a painful practice for other dependencies being upgrade with incompatible APIs.

          If we're only concerned about client side impact, then we should just upgrade guava and close this jira.

          We can not simply upgrade guava before shaded yarn/mr client works done. Otherwise, downstream projects will get stuck. Do I miss anything here?

          Show
          djp Junping Du added a comment - If a user wants Guava out of the classpath for the client side, why not just use the shaded client? I think we are working towards this direction. My understanding is YARN/MR client haven't finish shade practice. Isn't it? In either of those approaches folks are still out of luck on the server side. I thought that was the point of this jira? There shouldn't be too much concern on server side as application will have separated classloader so different version of guava should get loaded for different places. IMO, this JIRA approach is hardly being accepted not only because its own complexity but also open a painful practice for other dependencies being upgrade with incompatible APIs. If we're only concerned about client side impact, then we should just upgrade guava and close this jira. We can not simply upgrade guava before shaded yarn/mr client works done. Otherwise, downstream projects will get stuck. Do I miss anything here?
          Hide
          busbey Sean Busbey added a comment -

          we already have a shaded client for yarn/mr. there are some open issues against it, but fundamentally it works.

          Show
          busbey Sean Busbey added a comment - we already have a shaded client for yarn/mr. there are some open issues against it, but fundamentally it works.
          Hide
          busbey Sean Busbey added a comment -

          One gap that I'm not certain has a jira is that there isn't an IT that runs a MR job on the YARN minicluster provided by the shaded version of the minicluster jar.

          Show
          busbey Sean Busbey added a comment - One gap that I'm not certain has a jira is that there isn't an IT that runs a MR job on the YARN minicluster provided by the shaded version of the minicluster jar.
          Hide
          andrew.wang Andrew Wang added a comment -

          I've bumped this out to beta1, which is truly last call. Let's figure out our shading story by then.

          Show
          andrew.wang Andrew Wang added a comment - I've bumped this out to beta1, which is truly last call. Let's figure out our shading story by then.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Sorry for the delay.

          I would like to wrap up all opinions discussed here before moving forward. After that, I would like to continue to create a patch based on the approach.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Sorry for the delay. I would like to wrap up all opinions discussed here before moving forward. After that, I would like to continue to create a patch based on the approach.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Tsuyoshi Ozawa and everyone thanks for the work here.
          if we really want to get this into beta1, we'll need to really push harder.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Tsuyoshi Ozawa and everyone thanks for the work here. if we really want to get this into beta1, we'll need to really push harder.
          Hide
          djp Junping Du added a comment -

          if we really want to get this into beta1, we'll need to really push harder.

          We are still in discussion for a decent solution. From my understanding, this shouldn't be a real blocker for beta.

          Show
          djp Junping Du added a comment - if we really want to get this into beta1, we'll need to really push harder. We are still in discussion for a decent solution. From my understanding, this shouldn't be a real blocker for beta.
          Hide
          haibochen Haibo Chen added a comment -

          Following through the discussions, folks seem to prefer shading just client jars since we still want to use/upgrade guava in hadoop and we do not want to enforce the guava version on downstream projects.
          Junping Du Can you elaborate on why you think the shaded yarn/mr client is blocking the work here? Cann't we do them in parallel?
          To reduce build time, we could move the third-party module into an auxiliary hadoop project, like what HBase community is doing, so that we avoid shading every time hadoop is built. Thoughts?

          Show
          haibochen Haibo Chen added a comment - Following through the discussions, folks seem to prefer shading just client jars since we still want to use/upgrade guava in hadoop and we do not want to enforce the guava version on downstream projects. Junping Du Can you elaborate on why you think the shaded yarn/mr client is blocking the work here? Cann't we do them in parallel? To reduce build time, we could move the third-party module into an auxiliary hadoop project, like what HBase community is doing, so that we avoid shading every time hadoop is built. Thoughts?
          Hide
          stack stack added a comment -

          If it helps, here is what was done over in hbase to make it so we could upgrade guava, netty, protobuf, etc., w/o damage to downstreamers or having to use whatever hadoop et al. happened to have on the CLASSPATH

          • We made a little project hbase-thirdparty. Its only charge is providing mainline hbase with relocated popular libs such as guava and netty. The project comprises nought but poms (caveat some hacky patching of protobuf our project requires): https://github.com/apache/hbase-thirdparty The pull and relocation is moved out of the mainline hbase build.
          • We changed mainline hbase to use relocated versions of popular libs. This was mostly a case of changing imports from, for example, com.google.protobuf.Message to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message (an unfortunate decision a while back saddled us w/ the extra-long relocation prefix).
          • As part of the mainline build, we run com.google.code.maven-replacer-plugin to rewrite third-party references in generated code to instead refer to our relocated versions.

          Upside is we can update core libs whenever we wish. Should a lib turn problematic, we can add it to the relocated set. Downside is having to be sure we always refer to the relocated versions in code.

          While the pattern is straight-forward, the above project took a good while to implement mostly because infra is a bit shakey and our test suite has a host of flakies in it; verifying the test was failiing because it a flakey and not because of the relocation took a good while.

          If you want to do similar project in hadoop, I'd be game to help out.

          Show
          stack stack added a comment - If it helps, here is what was done over in hbase to make it so we could upgrade guava, netty, protobuf, etc., w/o damage to downstreamers or having to use whatever hadoop et al. happened to have on the CLASSPATH We made a little project hbase-thirdparty. Its only charge is providing mainline hbase with relocated popular libs such as guava and netty. The project comprises nought but poms (caveat some hacky patching of protobuf our project requires): https://github.com/apache/hbase-thirdparty The pull and relocation is moved out of the mainline hbase build. We changed mainline hbase to use relocated versions of popular libs. This was mostly a case of changing imports from, for example, com.google.protobuf.Message to org.apache.hadoop.hbase.shaded.com.google.protobuf.Message (an unfortunate decision a while back saddled us w/ the extra-long relocation prefix). As part of the mainline build, we run com.google.code.maven-replacer-plugin to rewrite third-party references in generated code to instead refer to our relocated versions. Upside is we can update core libs whenever we wish. Should a lib turn problematic, we can add it to the relocated set. Downside is having to be sure we always refer to the relocated versions in code. While the pattern is straight-forward, the above project took a good while to implement mostly because infra is a bit shakey and our test suite has a host of flakies in it; verifying the test was failiing because it a flakey and not because of the relocation took a good while. If you want to do similar project in hadoop, I'd be game to help out.
          Hide
          haibochen Haibo Chen added a comment -

          Thanks @stack for the insight on HBase's solution. Does HBase shade across all code base, instead of just in client modules?

          Downside is having to be sure we always refer to the relocated versions in code.

          Seems like maven enforcer can ban the unwanted non-relocated dependencies, manageable if provided that we are willing to reference the relocated versions in both server and client modules. If this does work, It seems to me that shading both server and client modules would be beneficial in the long run from maintenance point of view, even though what downstream projects need is only shaded clients. Thoughts?

          Does this approach address all of your concerns, Junping Du?

          Show
          haibochen Haibo Chen added a comment - Thanks @stack for the insight on HBase's solution. Does HBase shade across all code base, instead of just in client modules? Downside is having to be sure we always refer to the relocated versions in code. Seems like maven enforcer can ban the unwanted non-relocated dependencies, manageable if provided that we are willing to reference the relocated versions in both server and client modules. If this does work, It seems to me that shading both server and client modules would be beneficial in the long run from maintenance point of view, even though what downstream projects need is only shaded clients. Thoughts? Does this approach address all of your concerns, Junping Du ?
          Hide
          stack stack added a comment -

          Does HBase shade across all code base, instead of just in client modules?

          The whole code base.

          We offer a shaded client jar for those contained to our public facing client API. We've done a bad job evangelizing it up to this but intend to go at it with gusto with our next major release on out. This project should work for those confined to our client API but as with hadoop, hbase has facets other than that of the client API. This is when the story gets messy; i.e. plugins or clients reading/writing hbase files apart from an hbase instance. Shading our internals helps as we avoid the possibility of clashes. On our backend, we also are too-tightly tied to our upstream, a project we are working to undo, but we are not there yet. The internal shading helps here too.

          Using enforcer has been suggested over in our project. Will pass on our experience if anything noteworthy.

          Show
          stack stack added a comment - Does HBase shade across all code base, instead of just in client modules? The whole code base. We offer a shaded client jar for those contained to our public facing client API. We've done a bad job evangelizing it up to this but intend to go at it with gusto with our next major release on out. This project should work for those confined to our client API but as with hadoop, hbase has facets other than that of the client API. This is when the story gets messy; i.e. plugins or clients reading/writing hbase files apart from an hbase instance. Shading our internals helps as we avoid the possibility of clashes. On our backend, we also are too-tightly tied to our upstream, a project we are working to undo, but we are not there yet. The internal shading helps here too. Using enforcer has been suggested over in our project. Will pass on our experience if anything noteworthy.
          Hide
          haibochen Haibo Chen added a comment - - edited

          Shading the whole code base also makes more sense for hadoop IMO, given the fact that, as Andrew Wang pointed out in the comment above, downstream projects still depend on hadoop-hdfs, which is a server module. In YARN-7028 and YARN-6414, we also discovered that the guava version that hbase-client depends on can conflict with the guava version that hadoop uses, shading server modules (hbase-client is embedded in NM and RM) will help avoid that issue.

          Show
          haibochen Haibo Chen added a comment - - edited Shading the whole code base also makes more sense for hadoop IMO, given the fact that, as Andrew Wang pointed out in the comment above, downstream projects still depend on hadoop-hdfs, which is a server module. In YARN-7028 and YARN-6414 , we also discovered that the guava version that hbase-client depends on can conflict with the guava version that hadoop uses, shading server modules (hbase-client is embedded in NM and RM) will help avoid that issue.
          Hide
          stack stack added a comment -

          The particular example you cite goes away if hadoop embeds hbase2 since hbase will have relocated the problematic libs, but yeah, until then, embedding hbase has this problem as will other downstreamers

          Show
          stack stack added a comment - The particular example you cite goes away if hadoop embeds hbase2 since hbase will have relocated the problematic libs, but yeah, until then, embedding hbase has this problem as will other downstreamers
          Hide
          haibochen Haibo Chen added a comment -

          That's good to know. The TimelineServiceV2 backend is designed to be pluggable, so doing this will help other backend implementations, if any. Regardless, if we choose to shade server modules in addition to client modules, we don't need to think of whether we are in a client or server module because of the consistency and implementing the dependency enforcer will probably be easier.

          Show
          haibochen Haibo Chen added a comment - That's good to know. The TimelineServiceV2 backend is designed to be pluggable, so doing this will help other backend implementations, if any. Regardless, if we choose to shade server modules in addition to client modules, we don't need to think of whether we are in a client or server module because of the consistency and implementing the dependency enforcer will probably be easier.
          Hide
          templedf Daniel Templeton added a comment -

          We are still in discussion for a decent solution. From my understanding, this shouldn't be a real blocker for beta.

          Junping Du, classpath isolation has been one of the top level features for Hadoop 3.0 for as long as I can remember. If we do actually want to ship it in Hadoop 3.0, then we want to get the work in for beta1 if possible. Please help us by sharing what you see as open issues with the current proposal(s).

          Sounds like we have two options on the table:

          1. Shade just the client artifacts
          2. Shade everything

          (With all caveats and explanations as have already been covered above.) The idea of shading just the clients limits the impact, but it leaves open the door for unexpected downstream compatibility issues. It also gives us an inconsistent source base if we go the route of changing the imports to the relocated packages. Shading everything sounds safer, but is a larger change and may have a larger impact on the build process. It sounds to me like the 3rd party JAR should make the build process impact small. For these reasons, I'd would say to continue on the original path. Either way, though, let's make a call.

          Show
          templedf Daniel Templeton added a comment - We are still in discussion for a decent solution. From my understanding, this shouldn't be a real blocker for beta. Junping Du , classpath isolation has been one of the top level features for Hadoop 3.0 for as long as I can remember. If we do actually want to ship it in Hadoop 3.0, then we want to get the work in for beta1 if possible. Please help us by sharing what you see as open issues with the current proposal(s). Sounds like we have two options on the table: Shade just the client artifacts Shade everything (With all caveats and explanations as have already been covered above.) The idea of shading just the clients limits the impact, but it leaves open the door for unexpected downstream compatibility issues. It also gives us an inconsistent source base if we go the route of changing the imports to the relocated packages. Shading everything sounds safer, but is a larger change and may have a larger impact on the build process. It sounds to me like the 3rd party JAR should make the build process impact small. For these reasons, I'd would say to continue on the original path. Either way, though, let's make a call.

            People

            • Assignee:
              ozawa Tsuyoshi Ozawa
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              2 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated:

                Development