Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: timelineserver
    • Labels:
      None

      Description

      Recently phoenix has found some category-x dependencies in their build (PHOENIX-3084, PHOENIX-3091), which also showed some problems in HBase (HBASE-16260).

      Since the Timeline Server work brought in both of these as dependencies, we should make sure we don't have any cat-x dependencies either. From what I've seen in those projects, our choice of HBase version shouldn't be impacted but our Phoenix one is.

      Greping our current dependency list for the timeline server component shows some LGPL:

      ...
      [INFO]    net.sourceforge.findbugs:annotations:jar:1.3.2:compile
      ...
      

      I haven't checked the rest of the dependencies that have changed since HADOOP-12893 went in, so ATM I've filed this against YARN since that's where this one example came in.

      1. YARN-5433.04.patch
        27 kB
        Sangjin Lee
      2. YARN-5433.03.patch
        29 kB
        Sangjin Lee
      3. YARN-5433.02.patch
        3 kB
        Sangjin Lee
      4. YARN-5433.01.patch
        7 kB
        Sangjin Lee

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          fixing the findbugs annotations jar is easy, we just need to exclude it. We already have the clean room ALv2 reimplementation present.

          Show
          busbey Sean Busbey added a comment - fixing the findbugs annotations jar is easy, we just need to exclude it. We already have the clean room ALv2 reimplementation present.
          Hide
          andrew.wang Andrew Wang added a comment -

          Sangjin Lee could you or someone else who worked on TSv2 take this JIRA? It's a release blocker.

          Show
          andrew.wang Andrew Wang added a comment - Sangjin Lee could you or someone else who worked on TSv2 take this JIRA? It's a release blocker.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'll take a look. This is specifically about excluding findbugs (annotations), correct?

          Show
          sjlee0 Sangjin Lee added a comment - I'll take a look. This is specifically about excluding findbugs (annotations), correct?
          Hide
          andrew.wang Andrew Wang added a comment -

          findbugs is the most glaring issue, but we should check all the new dependencies brought in by the TSv2 merge. The discussion on HADOOP-12893 has the full list of methods, but it boils down to:

          • Run http://www.mojohaus.org/license-maven-plugin/ to look for potential bad maven dependencies. The spreadsheet at HADOOP-12893 addresses the false positives found by the plugin for existing libraries.
          • Check for any new third-party source code that's been copied in. This is mainly JS and native code.
          Show
          andrew.wang Andrew Wang added a comment - findbugs is the most glaring issue, but we should check all the new dependencies brought in by the TSv2 merge. The discussion on HADOOP-12893 has the full list of methods, but it boils down to: Run http://www.mojohaus.org/license-maven-plugin/ to look for potential bad maven dependencies. The spreadsheet at HADOOP-12893 addresses the false positives found by the plugin for existing libraries. Check for any new third-party source code that's been copied in. This is mainly JS and native code.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I looked into new dependencies that timelineservice modules add (~ 50 at the individual artifact level). I arrived at that list by looking at the dependencies from these modules and removing the ones that are marked as "done" in Andrew's spreadsheet.

          Most of them are ASLv2 licenses (e.g. HBase, Kerby, Tephra, Twill, etc.). I understand from HADOOP-12893 that ASLv2, BSD, and MIT license do not require separate mentions in our L&N. I then arrive at the following that still needs to be examined:

               (CDDL License) jsr311-api (javax.ws.rs:jsr311-api:1.1.1 - https://jsr311.dev.java.net)
               (Mozilla Public License Version 1.1) jamon-runtime (org.jamon:jamon-runtime:2.3.1 - http://www.jamon.org/jamon-runtime/)
               (Unknown license) sqlline (sqlline:sqlline:1.1.8 - no url defined)
          

          It appears that sqlline is a 3-clause BSD license (https://github.com/julianhyde/sqlline). I suppose we're OK with the JSR 311 (CDDL license)? How about the Mozilla license?

          Sean Busbey, I guess sqlline and jamon-runtime were looked at from the hbase and phoenix side of things? FYI, jsr311 is pulled in by jersey-core, jamon-runtime by hbase-server, and sqlline by phoenix-core.

          Show
          sjlee0 Sangjin Lee added a comment - I looked into new dependencies that timelineservice modules add (~ 50 at the individual artifact level). I arrived at that list by looking at the dependencies from these modules and removing the ones that are marked as "done" in Andrew's spreadsheet. Most of them are ASLv2 licenses (e.g. HBase, Kerby, Tephra, Twill, etc.). I understand from HADOOP-12893 that ASLv2, BSD, and MIT license do not require separate mentions in our L&N. I then arrive at the following that still needs to be examined: (CDDL License) jsr311-api (javax.ws.rs:jsr311-api:1.1.1 - https://jsr311.dev.java.net) (Mozilla Public License Version 1.1) jamon-runtime (org.jamon:jamon-runtime:2.3.1 - http://www.jamon.org/jamon-runtime/) (Unknown license) sqlline (sqlline:sqlline:1.1.8 - no url defined) It appears that sqlline is a 3-clause BSD license ( https://github.com/julianhyde/sqlline ). I suppose we're OK with the JSR 311 (CDDL license)? How about the Mozilla license? Sean Busbey , I guess sqlline and jamon-runtime were looked at from the hbase and phoenix side of things? FYI, jsr311 is pulled in by jersey-core, jamon-runtime by hbase-server, and sqlline by phoenix-core.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I also see the servlet API and the jsp API included from jetty:

               (CDDL 1.0) Glassfish Jasper (org.mortbay.jetty:jsp-2.1:6.1.14 - http://jetty.mortbay.org/project/modules/jsp-2.1)
               (CDDL 1.0) Servlet Specification 2.5 API (org.mortbay.jetty:servlet-api-2.5:6.1.14 - http://jetty.mortbay.org/project/modules/servlet-api-2.5)
          

          I'm not sure if they have been analyzed and addressed already.

          Show
          sjlee0 Sangjin Lee added a comment - I also see the servlet API and the jsp API included from jetty: (CDDL 1.0) Glassfish Jasper (org.mortbay.jetty:jsp-2.1:6.1.14 - http://jetty.mortbay.org/project/modules/jsp-2.1) (CDDL 1.0) Servlet Specification 2.5 API (org.mortbay.jetty:servlet-api-2.5:6.1.14 - http://jetty.mortbay.org/project/modules/servlet-api-2.5) I'm not sure if they have been analyzed and addressed already.
          Hide
          sjlee0 Sangjin Lee added a comment -

          A couple of jars with included source:

          • hbase-server-1.1.3.jar (contains javascript)
          • libthrift-0.9.0.jar (contains java source)
          Show
          sjlee0 Sangjin Lee added a comment - A couple of jars with included source: hbase-server-1.1.3.jar (contains javascript) libthrift-0.9.0.jar (contains java source)
          Hide
          sjlee0 Sangjin Lee added a comment -

          I have done a more comprehensive analysis, and am building a spreadsheet similar to HADOOP-12893: https://docs.google.com/spreadsheets/d/1D6aDHOUbQmF3SDtVj-3n4GhWlu6V8t4jD4z8HQivsq8/edit?usp=sharing

          So far almost all of the newly added dependencies appear to be ASLv.2, BSD, and MIT. It seems only ANTLR and sqlline may need to be added to NOTICES.txt, but I'm not 100% certain.

          cc Xiao Chen to see if he has feedback. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - I have done a more comprehensive analysis, and am building a spreadsheet similar to HADOOP-12893 : https://docs.google.com/spreadsheets/d/1D6aDHOUbQmF3SDtVj-3n4GhWlu6V8t4jD4z8HQivsq8/edit?usp=sharing So far almost all of the newly added dependencies appear to be ASLv.2, BSD, and MIT. It seems only ANTLR and sqlline may need to be added to NOTICES.txt, but I'm not 100% certain. cc Xiao Chen to see if he has feedback. Thanks!
          Hide
          sjlee0 Sangjin Lee added a comment -

          Attaching the first version of the patch.

          • excluded findbugs annotations from the dependencies
          • added notices for the new 3-clause BSD binaries (ANTLR, StringTemplate, and Sqlline)

          Any feedback on this patch or the analysis in the linked spreadsheet is welcome. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Attaching the first version of the patch. excluded findbugs annotations from the dependencies added notices for the new 3-clause BSD binaries (ANTLR, StringTemplate, and Sqlline) Any feedback on this patch or the analysis in the linked spreadsheet is welcome. Thanks!
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Sangjin Lee for working on this (and the ping).

          For HADOOP-12893, the spreadsheet has a 'generate.py' tab, containing the code used to generate the L&N (which license to skip L&N etc.). Probably an overkill here, but that's more accurate than my memory.

          So:

          Show
          xiaochen Xiao Chen added a comment - Thanks Sangjin Lee for working on this (and the ping). For HADOOP-12893 , the spreadsheet has a 'generate.py' tab, containing the code used to generate the L&N (which license to skip L&N etc.). Probably an overkill here, but that's more accurate than my memory. So: No need to add NOTICE for BSD here. See https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15284739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15284739 for details. MPL and CDDL are ok. We should add it to LICENSE.txt under corresponding license. There's already servlet-api 2.5 in there under CPPL1.0, so we can skip that. Didn't check all, but if there're any dependency with multiple licenses, we can choose the most friendly one.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the pointer Xiao Chen!

          I played with the python script, and generated some contents. I'll use the output to update the patch.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the pointer Xiao Chen ! I played with the python script, and generated some contents. I'll use the output to update the patch.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.2.

          Generated the output using the script provided by Xiao Chen.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.2. Generated the output using the script provided by Xiao Chen .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 9m 0s trunk passed
          +1 compile 9m 20s trunk passed
          +1 mvnsite 12m 11s trunk passed
          +1 mvneclipse 1m 13s trunk passed
          +1 javadoc 5m 18s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 9m 45s the patch passed
          +1 compile 7m 43s the patch passed
          +1 javac 7m 43s the patch passed
          +1 mvnsite 9m 55s the patch passed
          +1 mvneclipse 1m 4s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 4m 27s the patch passed
          -1 unit 105m 14s root in the patch failed.
          -1 asflicense 0m 24s The patch generated 2 ASF License warnings.
          177m 10s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache
          Timed out junit tests org.apache.hadoop.hdfs.protocol.TestAnnotations
            org.apache.hadoop.hdfs.TestFileChecksum
            org.apache.hadoop.cli.TestAclCLI
            org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete
            org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.TestFileConcurrentReader
            org.apache.hadoop.hdfs.TestDecommissionWithStriped
            org.apache.hadoop.hdfs.TestDistributedFileSystem
            org.apache.hadoop.hdfs.protocol.TestLocatedBlock
            org.apache.hadoop.hdfs.TestEncryptedTransfer
            org.apache.hadoop.hdfs.TestParallelUnixDomainRead



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835214/YARN-5433.01.patch
          JIRA Issue YARN-5433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux ae86d3b00f82 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 86c735b
          Default Java 1.8.0_101
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13512/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13512/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 9m 0s trunk passed +1 compile 9m 20s trunk passed +1 mvnsite 12m 11s trunk passed +1 mvneclipse 1m 13s trunk passed +1 javadoc 5m 18s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 9m 45s the patch passed +1 compile 7m 43s the patch passed +1 javac 7m 43s the patch passed +1 mvnsite 9m 55s the patch passed +1 mvneclipse 1m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 4m 27s the patch passed -1 unit 105m 14s root in the patch failed. -1 asflicense 0m 24s The patch generated 2 ASF License warnings. 177m 10s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestFsDatasetCache Timed out junit tests org.apache.hadoop.hdfs.protocol.TestAnnotations   org.apache.hadoop.hdfs.TestFileChecksum   org.apache.hadoop.cli.TestAclCLI   org.apache.hadoop.fs.contract.hdfs.TestHDFSContractDelete   org.apache.hadoop.hdfs.TestWriteConfigurationToDFS   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.TestFileConcurrentReader   org.apache.hadoop.hdfs.TestDecommissionWithStriped   org.apache.hadoop.hdfs.TestDistributedFileSystem   org.apache.hadoop.hdfs.protocol.TestLocatedBlock   org.apache.hadoop.hdfs.TestEncryptedTransfer   org.apache.hadoop.hdfs.TestParallelUnixDomainRead Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835214/YARN-5433.01.patch JIRA Issue YARN-5433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux ae86d3b00f82 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86c735b Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13512/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13512/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13512/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Sangjin Lee for the new rev!

          For MPL, since it's new, we need to put the whole MPL license text in there. The script can't handle it because we need to find the official license, and 80-char wrap it. I used http://appincredible.com/online/word-wrap/ to do the wrap in HADOOP-12893.

          Also, it seems the spreadsheet here has some already analyzed? == Y? entries (namely the glassfish ones). What's the plan on those? I think we should have the spreadsheet all Done==Y before resolving the jira.

          Show
          xiaochen Xiao Chen added a comment - Thanks Sangjin Lee for the new rev! For MPL, since it's new, we need to put the whole MPL license text in there. The script can't handle it because we need to find the official license, and 80-char wrap it. I used http://appincredible.com/online/word-wrap/ to do the wrap in HADOOP-12893 . Also, it seems the spreadsheet here has some already analyzed? == Y? entries (namely the glassfish ones). What's the plan on those? I think we should have the spreadsheet all Done==Y before resolving the jira.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 32s Maven dependency ordering for branch
          +1 mvninstall 6m 41s trunk passed
          +1 compile 6m 49s trunk passed
          +1 mvnsite 9m 19s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          +1 javadoc 4m 20s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 7m 12s the patch passed
          +1 compile 6m 51s the patch passed
          +1 javac 6m 51s the patch passed
          +1 mvnsite 9m 17s the patch passed
          +1 mvneclipse 1m 0s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 javadoc 4m 20s the patch passed
          -1 unit 92m 47s root in the patch failed.
          -1 asflicense 0m 21s The patch generated 2 ASF License warnings.
          152m 38s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
            hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
          Timed out junit tests org.apache.hadoop.hdfs.TestFileChecksum
            org.apache.hadoop.hdfs.TestDFSUtil
            org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
            org.apache.hadoop.hdfs.TestDFSStripedInputStream
            org.apache.hadoop.hdfs.server.balancer.TestBalancer
            org.apache.hadoop.hdfs.TestDecommissionWithStriped
            org.apache.hadoop.cli.TestAclCLIWithPosixAclInheritance
            org.apache.hadoop.hdfs.TestGetFileChecksum



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835233/YARN-5433.02.patch
          JIRA Issue YARN-5433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 4fad9a7fa3be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d88dca8
          Default Java 1.8.0_101
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13515/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13515/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 6m 41s trunk passed +1 compile 6m 49s trunk passed +1 mvnsite 9m 19s trunk passed +1 mvneclipse 0m 59s trunk passed +1 javadoc 4m 20s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 7m 12s the patch passed +1 compile 6m 51s the patch passed +1 javac 6m 51s the patch passed +1 mvnsite 9m 17s the patch passed +1 mvneclipse 1m 0s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 4m 20s the patch passed -1 unit 92m 47s root in the patch failed. -1 asflicense 0m 21s The patch generated 2 ASF License warnings. 152m 38s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Timed out junit tests org.apache.hadoop.hdfs.TestFileChecksum   org.apache.hadoop.hdfs.TestDFSUtil   org.apache.hadoop.hdfs.TestWriteConfigurationToDFS   org.apache.hadoop.hdfs.TestDFSStripedInputStream   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.TestDecommissionWithStriped   org.apache.hadoop.cli.TestAclCLIWithPosixAclInheritance   org.apache.hadoop.hdfs.TestGetFileChecksum Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835233/YARN-5433.02.patch JIRA Issue YARN-5433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 4fad9a7fa3be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d88dca8 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13515/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13515/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13515/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Some/most of the ones that I marked as "already analyzed" are the ones that were resolved in your original spreadsheet. I think a few more may need to be analyzed as part of this, and I'll do that.

          Regarding including the Mozilla Public License, this is what we're looking at: https://www.mozilla.org/en-US/MPL/1.1/

          I'm just not sure if this needs to be included in its entirety or only a certain section is relevant...

          Show
          sjlee0 Sangjin Lee added a comment - Some/most of the ones that I marked as "already analyzed" are the ones that were resolved in your original spreadsheet. I think a few more may need to be analyzed as part of this, and I'll do that. Regarding including the Mozilla Public License, this is what we're looking at: https://www.mozilla.org/en-US/MPL/1.1/ I'm just not sure if this needs to be included in its entirety or only a certain section is relevant...
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.3.

          Included the full MPL v.1.1.

          I analyzed a few more entries that were not previously analyzed by HADOOP-12893, and things like glassfish jasper (jsp) and servlet-api are actually redundant as they are Oracle/Sun API specs that were included in jetty (we already captured jsp and servlet-api under CDDL 1.0). So, there are no new entries.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.3. Included the full MPL v.1.1. I analyzed a few more entries that were not previously analyzed by HADOOP-12893 , and things like glassfish jasper (jsp) and servlet-api are actually redundant as they are Oracle/Sun API specs that were included in jetty (we already captured jsp and servlet-api under CDDL 1.0). So, there are no new entries.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the great work here, Sangjin Lee.

          I see the spreadsheet is done, and patch looks good. +1 pending the following nit:
          IIRC the 'EXHIBIT' section can be omitted, only the license is required. Somehow I couldn't find the source of this information anymore...

          Feel free to wait for potential comments from others, and merge this nit into the final patch.

          (For the record, MPL is category B. https://www.apache.org/legal/resolved#category-b)

          Show
          xiaochen Xiao Chen added a comment - Thanks for the great work here, Sangjin Lee . I see the spreadsheet is done, and patch looks good. +1 pending the following nit: IIRC the 'EXHIBIT' section can be omitted, only the license is required. Somehow I couldn't find the source of this information anymore... Feel free to wait for potential comments from others, and merge this nit into the final patch. (For the record, MPL is category B. https://www.apache.org/legal/resolved#category-b )
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 37s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 6m 55s trunk passed
          +1 mvnsite 9m 22s trunk passed
          +1 mvneclipse 1m 6s trunk passed
          +1 javadoc 4m 22s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 7m 18s the patch passed
          +1 compile 6m 58s the patch passed
          +1 javac 6m 58s the patch passed
          +1 mvnsite 9m 22s the patch passed
          +1 mvneclipse 1m 2s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 javadoc 4m 26s the patch passed
          -1 unit 101m 15s root in the patch failed.
          -1 asflicense 0m 21s The patch generated 2 ASF License warnings.
          161m 59s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
            org.apache.hadoop.hdfs.TestParallelRead
            org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            org.apache.hadoop.hdfs.TestPread
            org.apache.hadoop.hdfs.TestBlockMissingException
            org.apache.hadoop.hdfs.TestDFSClientRetries
            org.apache.hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835266/YARN-5433.03.patch
          JIRA Issue YARN-5433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 5438fefa4daf 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 287efff
          Default Java 1.8.0_101
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13518/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13518/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 6m 55s trunk passed +1 mvnsite 9m 22s trunk passed +1 mvneclipse 1m 6s trunk passed +1 javadoc 4m 22s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 7m 18s the patch passed +1 compile 6m 58s the patch passed +1 javac 6m 58s the patch passed +1 mvnsite 9m 22s the patch passed +1 mvneclipse 1m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 4m 26s the patch passed -1 unit 101m 15s root in the patch failed. -1 asflicense 0m 21s The patch generated 2 ASF License warnings. 161m 59s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestLargeBlockReport Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.TestParallelRead   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   org.apache.hadoop.hdfs.TestPread   org.apache.hadoop.hdfs.TestBlockMissingException   org.apache.hadoop.hdfs.TestDFSClientRetries   org.apache.hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835266/YARN-5433.03.patch JIRA Issue YARN-5433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 5438fefa4daf 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 287efff Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13518/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13518/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13518/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Posted patch v.4.

          Removed the exhibit portion of MPL.

          Thanks Xiao Chen for checking. I'll wait for a little bit for others to comment, and commit it unless there is an objection.

          Show
          sjlee0 Sangjin Lee added a comment - Posted patch v.4. Removed the exhibit portion of MPL. Thanks Xiao Chen for checking. I'll wait for a little bit for others to comment, and commit it unless there is an objection.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 6m 50s trunk passed
          +1 compile 6m 48s trunk passed
          +1 mvnsite 9m 14s trunk passed
          +1 mvneclipse 1m 2s trunk passed
          +1 javadoc 4m 17s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 7m 5s the patch passed
          +1 compile 6m 48s the patch passed
          +1 javac 6m 48s the patch passed
          +1 mvnsite 9m 7s the patch passed
          +1 mvneclipse 1m 0s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 javadoc 4m 16s the patch passed
          -1 unit 82m 8s root in the patch failed.
          -1 asflicense 0m 23s The patch generated 2 ASF License warnings.
          141m 38s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
            org.apache.hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
            org.apache.hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
            org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835368/YARN-5433.04.patch
          JIRA Issue YARN-5433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux fff145610caa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9cad3e2
          Default Java 1.8.0_101
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-unit-root.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13523/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13523/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 6m 50s trunk passed +1 compile 6m 48s trunk passed +1 mvnsite 9m 14s trunk passed +1 mvneclipse 1m 2s trunk passed +1 javadoc 4m 17s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 7m 5s the patch passed +1 compile 6m 48s the patch passed +1 javac 6m 48s the patch passed +1 mvnsite 9m 7s the patch passed +1 mvneclipse 1m 0s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 javadoc 4m 16s the patch passed -1 unit 82m 8s root in the patch failed. -1 asflicense 0m 23s The patch generated 2 ASF License warnings. 141m 38s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade   org.apache.hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand   org.apache.hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC   org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835368/YARN-5433.04.patch JIRA Issue YARN-5433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux fff145610caa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9cad3e2 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-unit-root.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13523/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13523/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13523/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed the patch. Thanks Xiao Chen, Andrew Wang, and Sean Busbey for your help and review.

          Show
          sjlee0 Sangjin Lee added a comment - Committed the patch. Thanks Xiao Chen , Andrew Wang , and Sean Busbey for your help and review.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10690 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10690/)
          YARN-5433. Audit dependencies for Category-X. Contributed by Sangjin (sjlee: rev f511cc89b66997e496f630bdd299d3068d43fd31)

          • (edit) LICENSE.txt
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10690 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10690/ ) YARN-5433 . Audit dependencies for Category-X. Contributed by Sangjin (sjlee: rev f511cc89b66997e496f630bdd299d3068d43fd31) (edit) LICENSE.txt (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml
          Hide
          sjlee0 Sangjin Lee added a comment -

          I also committed this to YARN-5355-branch-2. Any trunk changes for timeline service v.2 needs to be committed to YARN-5355-branch-2 as they are not picked up by rebase.

          Show
          sjlee0 Sangjin Lee added a comment - I also committed this to YARN-5355 -branch-2. Any trunk changes for timeline service v.2 needs to be committed to YARN-5355 -branch-2 as they are not picked up by rebase.

            People

            • Assignee:
              sjlee0 Sangjin Lee
              Reporter:
              busbey Sean Busbey
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development