Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14018

shaded jars of hadoop-client modules are missing hadoop's root LICENSE and NOTICE files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None

      Description

      In the 3.0.0-alpha2-RC0, hadoop-client-api-3.0.0-alpha2.jar misses LICENSE file, but it does have META-INF/NOTICE.

      1. licenses.txt
        9 kB
        Elek, Marton
      2. HADOOP-14018.002.patch
        6 kB
        Elek, Marton
      3. HADOOP-14018.002.patch
        6 kB
        Andrew Wang
      4. HADOOP-14018.001.patch
        3 kB
        Elek, Marton

        Issue Links

          Activity

          Hide
          elek Elek, Marton added a comment -

          I think the problem is that the ApacheLicenseResourceTransformer remove ALL of the licenses even the one from the original jar file. The safest method to add it again but it needs the usage of relative path to the LICENSE.txt in the project root.

          Show
          elek Elek, Marton added a comment - I think the problem is that the ApacheLicenseResourceTransformer remove ALL of the licenses even the one from the original jar file. The safest method to add it again but it needs the usage of relative path to the LICENSE.txt in the project root.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 13m 6s trunk passed
          +1 compile 0m 15s trunk passed
          +1 mvnsite 0m 38s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 6m 39s the patch passed
          +1 compile 0m 13s the patch passed
          +1 javac 0m 13s the patch passed
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 0m 21s the patch passed
          +1 unit 0m 9s hadoop-client-api in the patch passed.
          +1 unit 0m 9s hadoop-client-runtime in the patch passed.
          +1 unit 0m 9s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          25m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14018
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849004/HADOOP-14018.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux ead1a2495f23 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ce83205
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11500/testReport/
          modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11500/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 13m 6s trunk passed +1 compile 0m 15s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 39s trunk passed +1 javadoc 0m 29s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 6m 39s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 0m 21s the patch passed +1 unit 0m 9s hadoop-client-api in the patch passed. +1 unit 0m 9s hadoop-client-runtime in the patch passed. +1 unit 0m 9s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 25m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14018 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849004/HADOOP-14018.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux ead1a2495f23 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ce83205 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11500/testReport/ modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11500/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -
          $ ./dev-support/bin/verify-license-files 
          ERROR: hadoop-client-api-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found
          WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 6 LICENSE files (1 were valid)
          WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 3 NOTICE files (0 were valid)
          ERROR: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found
          WARNING: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: Found 2 LICENSE files (1 were valid)
          ERROR: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found
          

          Thanks Elek, Marton for the patch and John Zhuge for reporting.
          Appreciate the completeness to cover all hadoop-client modules although title only says hadoop-client-api.

          Allen had an enhanced tool in HADOOP-13374 to check for L&N, above is the result after applying the fix here. Could you take a look? Thanks.

          Show
          xiaochen Xiao Chen added a comment - $ ./dev-support/bin/verify-license-files ERROR: hadoop-client-api-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 6 LICENSE files (1 were valid) WARNING: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: Found 3 NOTICE files (0 were valid) ERROR: hadoop-client-minicluster-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found WARNING: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: Found 2 LICENSE files (1 were valid) ERROR: hadoop-client-runtime-3.0.0-alpha3-SNAPSHOT.jar: No valid NOTICE found Thanks Elek, Marton for the patch and John Zhuge for reporting. Appreciate the completeness to cover all hadoop-client modules although title only says hadoop-client-api. Allen had an enhanced tool in HADOOP-13374 to check for L&N, above is the result after applying the fix here. Could you take a look? Thanks.
          Hide
          elek Elek, Marton added a comment -

          I modified the patch to

          • exclude all the LICENCE and NOTICE files from the dependencies
          • Don't create one combined NOTICE based on the NOTICE files in the dependencies.
          • NOTICE.txt and LICENSE.txt is added from the project root dir.

          Tested with the script of HADOOP-13374. No WARNING/ERROR.

          According to the Licensing howto[1], all the relevant notices and licences should be added to the project NOTICE/LICENSE files. I also uploaded a report (for the reference) about which LICENCE/NOTICE files are removed during the shading.

          [1]: http://www.apache.org/dev/licensing-howto.html

          Show
          elek Elek, Marton added a comment - I modified the patch to exclude all the LICENCE and NOTICE files from the dependencies Don't create one combined NOTICE based on the NOTICE files in the dependencies. NOTICE.txt and LICENSE.txt is added from the project root dir. Tested with the script of HADOOP-13374 . No WARNING/ERROR. According to the Licensing howto [1] , all the relevant notices and licences should be added to the project NOTICE/LICENSE files. I also uploaded a report (for the reference) about which LICENCE/NOTICE files are removed during the shading. [1] : http://www.apache.org/dev/licensing-howto.html
          Hide
          andrew.wang Andrew Wang added a comment -

          Attaching same patch, test-patch is picking up licenses.txt instead

          Show
          andrew.wang Andrew Wang added a comment - Attaching same patch, test-patch is picking up licenses.txt instead
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 12m 52s trunk passed
          +1 compile 0m 16s trunk passed
          +1 mvnsite 0m 40s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 javadoc 0m 29s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 6m 28s the patch passed
          +1 compile 0m 13s the patch passed
          +1 javac 0m 13s the patch passed
          +1 mvnsite 0m 31s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 0m 21s the patch passed
          +1 unit 0m 9s hadoop-client-api in the patch passed.
          +1 unit 0m 8s hadoop-client-runtime in the patch passed.
          +1 unit 0m 10s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          24m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14018
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850281/HADOOP-14018.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux c740b178f280 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c6bae5
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11547/testReport/
          modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11547/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 12m 52s trunk passed +1 compile 0m 16s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 39s trunk passed +1 javadoc 0m 29s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 6m 28s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 0m 21s the patch passed +1 unit 0m 9s hadoop-client-api in the patch passed. +1 unit 0m 8s hadoop-client-runtime in the patch passed. +1 unit 0m 10s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 24m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14018 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850281/HADOOP-14018.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux c740b178f280 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c6bae5 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11547/testReport/ modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11547/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 16m 32s trunk passed
          +1 compile 0m 18s trunk passed
          +1 mvnsite 0m 47s trunk passed
          +1 mvneclipse 0m 53s trunk passed
          +1 javadoc 0m 37s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 8m 28s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          +1 mvnsite 0m 38s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 javadoc 0m 24s the patch passed
          +1 unit 0m 11s hadoop-client-api in the patch passed.
          +1 unit 0m 10s hadoop-client-runtime in the patch passed.
          +1 unit 0m 11s hadoop-client-minicluster in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          31m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-14018
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850281/HADOOP-14018.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 636e5197a6e6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c6bae5
          Default Java 1.8.0_121
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11548/testReport/
          modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11548/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 16m 32s trunk passed +1 compile 0m 18s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 53s trunk passed +1 javadoc 0m 37s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 8m 28s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 javadoc 0m 24s the patch passed +1 unit 0m 11s hadoop-client-api in the patch passed. +1 unit 0m 10s hadoop-client-runtime in the patch passed. +1 unit 0m 11s hadoop-client-minicluster in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 31m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-14018 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850281/HADOOP-14018.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 636e5197a6e6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c6bae5 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11548/testReport/ modules C: hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster U: hadoop-client-modules Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11548/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, built locally and also verified with the WIP L&N script. Will commit this shortly, thanks for the contribution Elek, Marton!

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, built locally and also verified with the WIP L&N script. Will commit this shortly, thanks for the contribution Elek, Marton !
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks again Elek, Marton!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks again Elek, Marton !
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Elek, Marton for the work, and Andrew Wang for the time to review/commit!

          Show
          xiaochen Xiao Chen added a comment - Thanks Elek, Marton for the work, and Andrew Wang for the time to review/commit!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11191 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11191/)
          HADOOP-14018. shaded jars of hadoop-client modules are missing hadoop's (wang: rev 258991dc5a3b13640834d1c8803f2eb840f8afea)

          • (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
          • (edit) hadoop-client-modules/hadoop-client-api/pom.xml
          • (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11191 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11191/ ) HADOOP-14018 . shaded jars of hadoop-client modules are missing hadoop's (wang: rev 258991dc5a3b13640834d1c8803f2eb840f8afea) (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml (edit) hadoop-client-modules/hadoop-client-api/pom.xml (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml

            People

            • Assignee:
              elek Elek, Marton
              Reporter:
              jzhuge John Zhuge
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development