Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5894

fixed license warning caused by de.ruedigermoeller:fst:jar:2.24

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: yarn
    • Labels:
      None

      Description

      The artifact de.ruedigermoeller:fst:jar:2.24, that ApplicationHistoryService depends on, shows its license being LGPL 2.1 in our license checking.

      1. YARN-5894.00.patch
        0.8 kB
        Haibo Chen
      2. YARN-5894.01.patch
        1.0 kB
        Haibo Chen

        Issue Links

          Activity

          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I checked and 2.41 and above versions of the artifact de.ruedigermoeller.fst are under the Apache license. Hadoop uses 2.24, which is LGPL.
          There is a conflict though:

          [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
          Failed while enforcing releasability the error(s) are [
          Dependency convergence error for org.javassist:javassist:3.21.0-GA paths to dependency are:
          +-org.apache.hadoop:hadoop-yarn-server-resourcemanager:3.0.0-alpha3-SNAPSHOT
            +-org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:3.0.0-alpha3-SNAPSHOT
              +-de.ruedigermoeller:fst:2.50
                +-org.javassist:javassist:3.21.0-GA
          and
          +-org.apache.hadoop:hadoop-yarn-server-resourcemanager:3.0.0-alpha3-SNAPSHOT
            +-org.apache.curator:curator-test:2.12.0
              +-org.javassist:javassist:3.18.1-GA
          ]
          
          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I checked and 2.41 and above versions of the artifact de.ruedigermoeller.fst are under the Apache license. Hadoop uses 2.24, which is LGPL. There is a conflict though: [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message: Failed while enforcing releasability the error(s) are [ Dependency convergence error for org.javassist:javassist:3.21.0-GA paths to dependency are: +-org.apache.hadoop:hadoop-yarn-server-resourcemanager:3.0.0-alpha3-SNAPSHOT +-org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:3.0.0-alpha3-SNAPSHOT +-de.ruedigermoeller:fst:2.50 +-org.javassist:javassist:3.21.0-GA and +-org.apache.hadoop:hadoop-yarn-server-resourcemanager:3.0.0-alpha3-SNAPSHOT +-org.apache.curator:curator-test:2.12.0 +-org.javassist:javassist:3.18.1-GA ]
          Hide
          rchiang Ray Chiang added a comment -

          Jonathan Eagles, it looks like this was brought in as part of YARN-3448. Did we get some special permission for this license issue? It seems like this would need to be corrected before final 3.0 release.

          Show
          rchiang Ray Chiang added a comment - Jonathan Eagles , it looks like this was brought in as part of YARN-3448 . Did we get some special permission for this license issue? It seems like this would need to be corrected before final 3.0 release.
          Hide
          haibochen Haibo Chen added a comment -

          The patch upgrades fst to 2.50 which comes with Apache license. org.javassist:javassist has been excluded based on https://github.com/RuedigerMoeller/fast-serialization/blob/aceaad0075b2e1ef796597a1098aeb39fbea7fc9/pom.xml#L134. The exclusion did not cause tests to fail.

          Show
          haibochen Haibo Chen added a comment - The patch upgrades fst to 2.50 which comes with Apache license. org.javassist:javassist has been excluded based on https://github.com/RuedigerMoeller/fast-serialization/blob/aceaad0075b2e1ef796597a1098aeb39fbea7fc9/pom.xml#L134 . The exclusion did not cause tests to fail.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 35s trunk passed
          +1 compile 0m 22s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 22s the patch passed
          +1 javac 0m 22s the patch passed
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 15s the patch passed
          +1 unit 2m 48s hadoop-yarn-server-applicationhistoryservice in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          22m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5894
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865014/YARN-5894.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 03bda33dcd33 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 475f933
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15737/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15737/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 35s trunk passed +1 compile 0m 22s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 16s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 15s the patch passed +1 unit 2m 48s hadoop-yarn-server-applicationhistoryservice in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 22m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5894 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865014/YARN-5894.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 03bda33dcd33 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 475f933 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15737/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice Console output https://builds.apache.org/job/PreCommit-YARN-Build/15737/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          I did a diff of the jars before and after apply the patch. It looks like upgrading FST to 2.50 also:

          • Adds java-util 1.9.0
          • Adds json-io 2.5.1
          • Upgrades objenesis 2.1 to 2.5.1
          < ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/fst-2.24.jar
          ---
          > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/fst-2.50.jar
          1379a1382
          > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/java-util-1.9.0.jar
          1404a1408
          > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/json-io-2.5.1.jar
          1432c1436
          < ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/objenesis-2.1.jar
          ---
          > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/objenesis-2.5.1.jar
          

          Looks like the added libraries are Apache Licensed. Can you just verify that there's no problem if we upgrade objenesis?

          Show
          rkanter Robert Kanter added a comment - I did a diff of the jars before and after apply the patch. It looks like upgrading FST to 2.50 also: Adds java-util 1.9.0 Adds json-io 2.5.1 Upgrades objenesis 2.1 to 2.5.1 < ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/fst-2.24.jar --- > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/fst-2.50.jar 1379a1382 > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/java-util-1.9.0.jar 1404a1408 > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/json-io-2.5.1.jar 1432c1436 < ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/objenesis-2.1.jar --- > ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/lib/objenesis-2.5.1.jar Looks like the added libraries are Apache Licensed. Can you just verify that there's no problem if we upgrade objenesis?
          Hide
          haibochen Haibo Chen added a comment -

          Looks like objenesis is needed only for android per https://github.com/RuedigerMoeller/fast-serialization/blob/aceaad0075b2e1ef796597a1098aeb39fbea7fc9/pom.xml#L141

          I am going to exclude that as well, and see if it causes any problem.

          Show
          haibochen Haibo Chen added a comment - Looks like objenesis is needed only for android per https://github.com/RuedigerMoeller/fast-serialization/blob/aceaad0075b2e1ef796597a1098aeb39fbea7fc9/pom.xml#L141 I am going to exclude that as well, and see if it causes any problem.
          Hide
          haibochen Haibo Chen added a comment -

          Updated the patch to also exclude objenesis

          Show
          haibochen Haibo Chen added a comment - Updated the patch to also exclude objenesis
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 42s trunk passed
          +1 compile 0m 24s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 javadoc 0m 17s trunk passed
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 0m 12s the patch passed
          +1 unit 2m 50s hadoop-yarn-server-applicationhistoryservice in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          22m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5894
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865189/YARN-5894.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 23444e724a67 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8b5f2c3
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15752/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15752/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 42s trunk passed +1 compile 0m 24s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 17s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 12s the patch passed +1 unit 2m 50s hadoop-yarn-server-applicationhistoryservice in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 22m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5894 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865189/YARN-5894.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 23444e724a67 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b5f2c3 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15752/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice Console output https://builds.apache.org/job/PreCommit-YARN-Build/15752/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          +1

          Show
          rkanter Robert Kanter added a comment - +1
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Haibo Chen. Committed to trunk!

          Show
          rkanter Robert Kanter added a comment - Thanks Haibo Chen . Committed to trunk!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11645/)
          YARN-5894. fixed license warning caused by (rkanter: rev 371b6467dcee8517ef0d49b9f391302a6d9ed648)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11645 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11645/ ) YARN-5894 . fixed license warning caused by (rkanter: rev 371b6467dcee8517ef0d49b9f391302a6d9ed648) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
          Hide
          djp Junping Du added a comment -

          I think branch-2 and branch-2.8.1 need this fix also. Robert Kanter and Haibo Chen, could you backport to these branches also?

          Show
          djp Junping Du added a comment - I think branch-2 and branch-2.8.1 need this fix also. Robert Kanter and Haibo Chen , could you backport to these branches also?
          Hide
          rkanter Robert Kanter added a comment -

          Sure thing - committed to branch-2, branch-2.8, and branch-2.8.1!

          Show
          rkanter Robert Kanter added a comment - Sure thing - committed to branch-2, branch-2.8, and branch-2.8.1!
          Hide
          djp Junping Du added a comment -

          Thanks Robert!

          Show
          djp Junping Du added a comment - Thanks Robert!
          Hide
          jlowe Jason Lowe added a comment -

          This change is causing jackson-core-2.2.3 to be pulled in as a dependency for Hadoop 2.8.1, and that is breaking jobs that ran fine on Hadoop 2.8.0. See YARN-6628.

          Show
          jlowe Jason Lowe added a comment - This change is causing jackson-core-2.2.3 to be pulled in as a dependency for Hadoop 2.8.1, and that is breaking jobs that ran fine on Hadoop 2.8.0. See YARN-6628 .
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              haibochen Haibo Chen
              Reporter:
              haibochen Haibo Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development