Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13780

LICENSE/NOTICE are out of date for source artifacts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: common
    • Labels:
      None

      Description

      we need to perform a check that all of our bundled works are properly accounted for in our LICENSE/NOTICE files.

      At a minimum, it looks like HADOOP-10075 introduced some changes that have not been accounted for.

      e.g. the jsTree plugin found at hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js does not show up in LICENSE.txt to (a) indicate that we're redistributing it under the MIT option and (b) give proper citation of the original copyright holder per ASF policy.

      1. HADOOP-13780.01.patch
        21 kB
        Xiao Chen
      2. HADOOP-13780.02.patch
        21 kB
        Xiao Chen
      3. HADOOP-13780.03.patch
        59 kB
        Xiao Chen
      4. HADOOP-13780.03-with-scripts.patch
        68 kB
        Xiao Chen
      5. HADOOP-13780.04.patch
        78 kB
        Xiao Chen
      6. HADOOP-13780.04-with-scripts.patch
        103 kB
        Xiao Chen
      7. HADOOP-13780.05.patch
        80 kB
        Xiao Chen
      8. HADOOP-13780.06.patch
        80 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          busbey Sean Busbey added a comment -

          Important to note that the jsTree example is not meant to be exhaustive; I did not look to see what else wasn't updated I just randomly searched for a copyright string. I also have not yet looked to see if the binary bundlings properly account for the update (see HBASE-12894 for where folks over in HBase are checking for the same).

          Show
          busbey Sean Busbey added a comment - Important to note that the jsTree example is not meant to be exhaustive; I did not look to see what else wasn't updated I just randomly searched for a copyright string. I also have not yet looked to see if the binary bundlings properly account for the update (see HBASE-12894 for where folks over in HBase are checking for the same).
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Updated the list of bundled jars which was originally created for HADOOP-12893.
          https://gist.github.com/aajisaka/6f61ae083770739d57720745bcb12f0d/revisions

          Show
          ajisakaa Akira Ajisaka added a comment - Updated the list of bundled jars which was originally created for HADOOP-12893 . https://gist.github.com/aajisaka/6f61ae083770739d57720745bcb12f0d/revisions
          Hide
          rkanter Robert Kanter added a comment -

          HADOOP-10075 didn't add any new js files; they were all there this entire time, but many of them were only gzip versions (i.e. jquery.jstree.js.gz --> jquery.jstree.js). I guess we've been missing this for a long time then.

          Show
          rkanter Robert Kanter added a comment - HADOOP-10075 didn't add any new js files; they were all there this entire time, but many of them were only gzip versions (i.e. jquery.jstree.js.gz --> jquery.jstree.js). I guess we've been missing this for a long time then.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Sean Busbey for reporting this. I'd like to take a shot at this one to move alpha2 forward.

          It seems more things are added since Akira's last update (188 lines now in my run today https://gist.github.com/xiao-chen/336b64b1b17e8813fd5b980013ac7eb4)

          I plan to do the following things here:

          1. Fix the diff in L&N since HADOOP-12893, in a similar way.
          2. Manually fix the jstree stuff, and others turned out missing. Looks like this has to be manual, without some sophisticated tooling. As Robert said, HADOOP-10075 only extracted that jquery.jstree.js.gz, which was committed by.... YARN-1.
          3. Add a way to verify this in pre-commit, so this work in the future will be upfront.

          1 and 2 should unblock the release, 3 would make our lives easier.

          Show
          xiaochen Xiao Chen added a comment - Thanks Sean Busbey for reporting this. I'd like to take a shot at this one to move alpha2 forward. It seems more things are added since Akira's last update (188 lines now in my run today https://gist.github.com/xiao-chen/336b64b1b17e8813fd5b980013ac7eb4 ) I plan to do the following things here: Fix the diff in L&N since HADOOP-12893 , in a similar way. Manually fix the jstree stuff, and others turned out missing. Looks like this has to be manual, without some sophisticated tooling. As Robert said, HADOOP-10075 only extracted that jquery.jstree.js.gz, which was committed by.... YARN-1 . Add a way to verify this in pre-commit, so this work in the future will be upfront. 1 and 2 should unblock the release, 3 would make our lives easier.
          Hide
          xiaochen Xiao Chen added a comment -

          Regarding #1, thanks Akira Ajisaka for the commands from HADOOP-12893, I built a new output at https://gist.github.com/xiao-chen/6131ec9718ec4b1af286f048bd714c6f . Also looked at Apache Rat which seems too naive, and Apache Whisker which isn't documented clear enough (to me).

          Quick look at #2:

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js is actually noted in LICENSE! It is named with .gz extension. Since HADOOP-10075 removed the .gz and left the extracted .js, I think updating the name and move it to MIT License section in our LICENSE should suffice. This is legal since the header of that file says it's MIT, and apache need not that to be in the NOTICE.

          Bad news is, those js, css or anything outside of a maven dependency isn't checked by the tool.

          Show
          xiaochen Xiao Chen added a comment - Regarding #1, thanks Akira Ajisaka for the commands from HADOOP-12893 , I built a new output at https://gist.github.com/xiao-chen/6131ec9718ec4b1af286f048bd714c6f . Also looked at Apache Rat which seems too naive, and Apache Whisker which isn't documented clear enough (to me). Quick look at #2: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jt/jquery.jstree.js is actually noted in LICENSE! It is named with .gz extension. Since HADOOP-10075 removed the .gz and left the extracted .js, I think updating the name and move it to MIT License section in our LICENSE should suffice. This is legal since the header of that file says it's MIT, and apache need not that to be in the NOTICE. Bad news is, those js , css or anything outside of a maven dependency isn't checked by the tool.
          Hide
          xiaochen Xiao Chen added a comment -

          Update for #1: got a parsed result at https://docs.google.com/spreadsheets/d/1jpeVlwydkgM01FNW4GPdAgzch5kuLC8ka09yiewKy7w/edit#gid=1885055871.
          We had 211 rows when doing HADOOP-12893, now it is 356. I don't see any disallowed license at the first glance, but will go through them in details. If anyone knows any lawyer super weapon to automate/shortcut this, please shout.

          Show
          xiaochen Xiao Chen added a comment - Update for #1: got a parsed result at https://docs.google.com/spreadsheets/d/1jpeVlwydkgM01FNW4GPdAgzch5kuLC8ka09yiewKy7w/edit#gid=1885055871 . We had 211 rows when doing HADOOP-12893 , now it is 356. I don't see any disallowed license at the first glance, but will go through them in details. If anyone knows any lawyer super weapon to automate/shortcut this, please shout.
          Hide
          xiaochen Xiao Chen added a comment -

          In the faith of unblocking 3.0.0-alpha2 release, how do people feel about doing #1 and #2 from my above comment in this jira, and defer the automation #3 to another jira? #1 is almost done, and #2 shouldn't be too hard. So should be able to post a patch this week.

          I have some local nasty scripts to sort of automate this, with some things that need manual inspection. However even myself feel those scripts are not to the standards... don't want them to block our mighty hadoop release.

          Show
          xiaochen Xiao Chen added a comment - In the faith of unblocking 3.0.0-alpha2 release, how do people feel about doing #1 and #2 from my above comment in this jira, and defer the automation #3 to another jira? #1 is almost done, and #2 shouldn't be too hard. So should be able to post a patch this week. I have some local nasty scripts to sort of automate this, with some things that need manual inspection. However even myself feel those scripts are not to the standards... don't want them to block our mighty hadoop release.
          Hide
          busbey Sean Busbey added a comment -

          yeah that sounds like a good plan. Anything we come up with for precommit checking is going to be a heuristic that can always be improved.

          Show
          busbey Sean Busbey added a comment - yeah that sounds like a good plan. Anything we come up with for precommit checking is going to be a heuristic that can always be improved.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Sean for the comment.

          I have finished up a first draft of #1, shown in the 'Dependencies' tab of this jira's linked spreadsheet. Will work on closing the final gaps, and start on #2.

          Among those dependencies:

          • jdiff is LGPL but according to HADOOP-12893, it's not bundled so we're good.
          • ldapsdk is new, I did a quick search in pom but didn't find any. Will look more.
          • JSON needs some help: Akira Ajisaka Andrew Wang, apologize for my fading memory, but do you recall what was done for that in HADOOP-12893? (Searched the jira but no mention from the comments, and in the spreadsheet it's marked Done? == N... I seem to remember all things are done when we posted patches/resolved that jira.) Anyways, bundled? is also N, so I'm guessing that's the reason this is omitted at that time.

          Ping me if anyone wants edit perm to the spreadsheet. Note that the Dependencies and parsed tabs are totally script-generated, and are supposed to be replaced in later runs. In case anyone is curious, here's how to (nastily) generate:

          xiao-MBP:license xiao$ cat step1.sh 
          #!/bin/sh -x
          
          # First save spreadsheet to local:
          # 'Licenses' tab to licenses.tsv
          # 'Overrides' tab to overrides.tsv
          # 'parse.py script' tab to parse.py
          # 'standardize.py' tab to standardize.py
          # 'generate.py script' tab to generate.py
          
          mvn license:aggregate-add-third-party
          
          OUTPUT_DIR=~/Downloads/license/
          cp target/generated-sources/license/THIRD-PARTY.txt $OUTPUT_DIR
          
          
          xiao-MBP:license xiao$ cat step2.sh 
          #!/bin/sh -x
          
          python parse.py > parsed.tsv
          xiao-MBP:license xiao$ cat step3.sh 
          #!/bin/sh -x
          
          python standardize.py
          # will generate a standardized.tsv, which is the 'Dependencies' tab in the spreadsheet.
          
          Show
          xiaochen Xiao Chen added a comment - Thanks Sean for the comment. I have finished up a first draft of #1, shown in the 'Dependencies' tab of this jira's linked spreadsheet. Will work on closing the final gaps, and start on #2. Among those dependencies: jdiff is LGPL but according to HADOOP-12893 , it's not bundled so we're good. ldapsdk is new, I did a quick search in pom but didn't find any. Will look more. JSON needs some help: Akira Ajisaka Andrew Wang , apologize for my fading memory, but do you recall what was done for that in HADOOP-12893 ? (Searched the jira but no mention from the comments, and in the spreadsheet it's marked Done? == N... I seem to remember all things are done when we posted patches/resolved that jira.) Anyways, bundled? is also N, so I'm guessing that's the reason this is omitted at that time. Ping me if anyone wants edit perm to the spreadsheet. Note that the Dependencies and parsed tabs are totally script-generated, and are supposed to be replaced in later runs. In case anyone is curious, here's how to (nastily) generate: xiao-MBP:license xiao$ cat step1.sh #!/bin/sh -x # First save spreadsheet to local: # 'Licenses' tab to licenses.tsv # 'Overrides' tab to overrides.tsv # 'parse.py script' tab to parse.py # 'standardize.py' tab to standardize.py # 'generate.py script' tab to generate.py mvn license:aggregate-add-third-party OUTPUT_DIR=~/Downloads/license/ cp target/generated-sources/license/THIRD-PARTY.txt $OUTPUT_DIR xiao-MBP:license xiao$ cat step2.sh #!/bin/sh -x python parse.py > parsed.tsv xiao-MBP:license xiao$ cat step3.sh #!/bin/sh -x python standardize.py # will generate a standardized.tsv, which is the 'Dependencies' tab in the spreadsheet.
          Hide
          andrew.wang Andrew Wang added a comment -

          I think JSON is JSON.org, which should be covered based on notes in HADOOP-13794.

          I'm sad if we've somehow added over 100 dependencies in a few months since the last L&N update, but I think I massaged the list last time. We can remove our own Apache Hadoop deps from that list for instance, and there are entries for Apache DS and Maven that can be collapsed.

          Show
          andrew.wang Andrew Wang added a comment - I think JSON is JSON.org, which should be covered based on notes in HADOOP-13794 . I'm sad if we've somehow added over 100 dependencies in a few months since the last L&N update, but I think I massaged the list last time. We can remove our own Apache Hadoop deps from that list for instance, and there are entries for Apache DS and Maven that can be collapsed.
          Hide
          xiaochen Xiao Chen added a comment -

          Cool, will leave JSON out of the L&N since it's test-only, and have HADOOP-13794 deal with it.

          Surprised me on the dependency growth too, but as you said they won't necessarily all be listed in the LICENSE/NOTICE. Also this includes transitive deps, for example as I find out the ldapsdk in my above comment is from this, so should be good too:

          $ mvn license:aggregate-add-third-party -X -e
          ...
          [INFO] Forking Apache Hadoop Auth 3.0.0-alpha2-SNAPSHOT
          ...
          [DEBUG] org.apache.hadoop:hadoop-auth:jar:3.0.0-alpha2-SNAPSHOT
          ...
          [DEBUG]    org.apache.directory.server:apacheds-server-integ:jar:2.0.0-M21:test
          ...
          [DEBUG]       ldapsdk:ldapsdk:jar:4.1:test
          
          Show
          xiaochen Xiao Chen added a comment - Cool, will leave JSON out of the L&N since it's test-only, and have HADOOP-13794 deal with it. Surprised me on the dependency growth too, but as you said they won't necessarily all be listed in the LICENSE/NOTICE. Also this includes transitive deps, for example as I find out the ldapsdk in my above comment is from this, so should be good too: $ mvn license:aggregate-add-third-party -X -e ... [INFO] Forking Apache Hadoop Auth 3.0.0-alpha2-SNAPSHOT ... [DEBUG] org.apache.hadoop:hadoop-auth:jar:3.0.0-alpha2-SNAPSHOT ... [DEBUG] org.apache.directory.server:apacheds-server-integ:jar:2.0.0-M21:test ... [DEBUG] ldapsdk:ldapsdk:jar:4.1:test
          Hide
          xiaochen Xiao Chen added a comment -

          Attaching a patch 1 that takes care of everything HADOOP-12893 has done for alpha-1.

          I also have the automated scripts at https://github.com/xiao-chen/hadoop/tree/13780/dev-support/license , step1-step4 should give 2 files notices and licenses (instructions in step1). Merging to current L&N files are manual.

          Only thing left for this jira is look for (compressed) files like jstree, and include those as well. I think this is just a matter of time, and should be less than 1/2 day of work. (step5.sh will do this, currently not working).

          Any review/comments appreciated!

          Show
          xiaochen Xiao Chen added a comment - Attaching a patch 1 that takes care of everything HADOOP-12893 has done for alpha-1. I also have the automated scripts at https://github.com/xiao-chen/hadoop/tree/13780/dev-support/license , step1-step4 should give 2 files notices and licenses (instructions in step1). Merging to current L&N files are manual. Only thing left for this jira is look for (compressed) files like jstree, and include those as well. I think this is just a matter of time, and should be less than 1/2 day of work. (step5.sh will do this, currently not working). Any review/comments appreciated!
          Hide
          xiaochen Xiao Chen added a comment -

          Attaching patch 2, ready for review.

          Done grepping and checking all files with copyright|©. Looks like we did a good job on those js/css initially. Parsed result in spreadsheet's "copyright files" tab. Just had to rename a couple of things.

          Checked in the step5.sh to my branch too in case anyone want to take a look. Manually removed local files and pasted the left 23 lines into the spreadsheet.

          Show
          xiaochen Xiao Chen added a comment - Attaching patch 2, ready for review. Done grepping and checking all files with copyright|© . Looks like we did a good job on those js/css initially. Parsed result in spreadsheet's "copyright files" tab. Just had to rename a couple of things. Checked in the step5.sh to my branch too in case anyone want to take a look. Manually removed local files and pasted the left 23 lines into the spreadsheet.
          Hide
          busbey Sean Busbey added a comment -

          can someone give me the reasoning on why we don't want our in-progress best efforts at scripts for this workflow included?

          Show
          busbey Sean Busbey added a comment - can someone give me the reasoning on why we don't want our in-progress best efforts at scripts for this workflow included?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          0m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13780
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844149/HADOOP-13780.02.patch
          Optional Tests asflicense
          uname Linux ca23f55ebf96 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f678080
          modules C: . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11306/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 0m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13780 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844149/HADOOP-13780.02.patch Optional Tests asflicense uname Linux ca23f55ebf96 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f678080 modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11306/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          I was thinking they're not elegant enough for the hadoop code base, and would need some extra reviews, so not included here to prevent distraction from L&N themselves. Also there're still some manual steps (e.g. merging the generated L&N into current, checking what NOTICE should a dependency need etc). But if all-inclusive is desired here, I can try.

          Show
          xiaochen Xiao Chen added a comment - I was thinking they're not elegant enough for the hadoop code base, and would need some extra reviews, so not included here to prevent distraction from L&N themselves. Also there're still some manual steps (e.g. merging the generated L&N into current, checking what NOTICE should a dependency need etc). But if all-inclusive is desired here, I can try.
          Hide
          xiaochen Xiao Chen added a comment -

          Add a rebased patch 3 - and also ran the scripts to reflect the current state.

          Also adding the best-effort scripts on top of patch 3 - at least they're self-documenting now. Notably there are still some manual steps and improvements to be done. Lawyer's road isn't easy.

          Show
          xiaochen Xiao Chen added a comment - Add a rebased patch 3 - and also ran the scripts to reflect the current state. Also adding the best-effort scripts on top of patch 3 - at least they're self-documenting now. Notably there are still some manual steps and improvements to be done. Lawyer's road isn't easy.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          0m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13780
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845336/HADOOP-13780.03.patch
          Optional Tests asflicense
          uname Linux c6fa366af976 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6938b67
          modules C: . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11340/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 0m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13780 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845336/HADOOP-13780.03.patch Optional Tests asflicense uname Linux c6fa366af976 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6938b67 modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11340/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for your hard work on this Xiao! +1 LGTM to unblock the release, though we need a follow-on to improve the scripts. I'm sure these are already on your todo list, but a few thoughts along those lines:

          • We should try to remove the dependency on the externally managed GDoc. Checking in an exported sqlite DB or some csvs would be an improvement.
          • generate.py is still in the spreadsheet
          • the manual merge step is unfortunate, ideally everything is fully-generated by a single script and input data.

          Also, did we ever file a JIRA to do a precommit check?

          Show
          andrew.wang Andrew Wang added a comment - Thanks for your hard work on this Xiao! +1 LGTM to unblock the release, though we need a follow-on to improve the scripts. I'm sure these are already on your todo list, but a few thoughts along those lines: We should try to remove the dependency on the externally managed GDoc. Checking in an exported sqlite DB or some csvs would be an improvement. generate.py is still in the spreadsheet the manual merge step is unfortunate, ideally everything is fully-generated by a single script and input data. Also, did we ever file a JIRA to do a precommit check?
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew, filed 2 jiras linked here, for pre-commit and for automation.

          Agree on getting rid of the gdoc - we can just use tsvs but for this jira, the gdoc is the one place to include all of them. Interesting idea about sqlite db, will play with it.

          The fully-automatic is possible, but more work needed than current state (what I do manually now):

          • Those raw files (js/css/etc.) needs a doc to get managed, and merged
          • Need to add a new entry for the overrides so we can intentionally ignore some (jdiff, json, ldapsdk as we found out so far)
            Once those are done, will need a wiki/instruction page to use it.
          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew, filed 2 jiras linked here, for pre-commit and for automation. Agree on getting rid of the gdoc - we can just use tsvs but for this jira, the gdoc is the one place to include all of them. Interesting idea about sqlite db, will play with it. The fully-automatic is possible, but more work needed than current state (what I do manually now): Those raw files (js/css/etc.) needs a doc to get managed, and merged Need to add a new entry for the overrides so we can intentionally ignore some (jdiff, json, ldapsdk as we found out so far) Once those are done, will need a wiki/instruction page to use it.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Andrew Wang for the +1! I plan to commit this on Thursday if no objections by then.

          Show
          xiaochen Xiao Chen added a comment - Thanks Andrew Wang for the +1! I plan to commit this on Thursday if no objections by then.
          Hide
          busbey Sean Busbey added a comment -

          The entries for the bundled HBase libraries is slightly incorrect.

          in LICENSE:

          • The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012.
          • There's no entry for the bundled Orca Logo from the HBase Server jar. It's mentioned in NOTICE, but LICENSE should have a complete reference for the CC-BY 3.0 license (found in LICENSE from the hbase-server-1.1.3.jar)

          in NOTICE:

          • I don't see any actual inclusion of HBase Shell 1.1.3, HBase IT Tests 1.1.3, or HBase Testing Utility 1.1.3 artifacts. I'm not sure if this is an oversight in the constructed NOTICE or in the bin distribution tarball.
          Show
          busbey Sean Busbey added a comment - The entries for the bundled HBase libraries is slightly incorrect. in LICENSE: The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012. There's no entry for the bundled Orca Logo from the HBase Server jar. It's mentioned in NOTICE, but LICENSE should have a complete reference for the CC-BY 3.0 license (found in LICENSE from the hbase-server-1.1.3.jar) in NOTICE: I don't see any actual inclusion of HBase Shell 1.1.3, HBase IT Tests 1.1.3, or HBase Testing Utility 1.1.3 artifacts. I'm not sure if this is an oversight in the constructed NOTICE or in the bin distribution tarball.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot for the review Sean Busbey!

          The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012.

          jquery actually has been like this long ago... Looking at http://www.apache.org/dev/licensing-howto.html#permissive-deps , is the year required? I'm guessing the current way is written without year because the first 2 are 2005, 2013 and the last is 2012:

          ...
          hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-1.10.2.min.js
          hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery
          --------------------------------------------------------------------------------

          Copyright jQuery Foundation and other contributors, https://jquery.org/
          ...

          I'll be happy to update accordingly, but wanted to make sure - apache licensing seems is saying 'add a pointer', 'short note summarizing', and the example there didn't even mention copyright...

          There's no entry for the bundled Orca Logo from the HBase Server jar. It's mentioned in NOTICE, but LICENSE should have a complete reference for the CC-BY 3.0 license (found in LICENSE from the hbase-server-1.1.3.jar)

          Copied from there and added to hadoop LICENSE.

          I don't see any actual inclusion of HBase Shell 1.1.3, HBase IT Tests 1.1.3, or HBase Testing Utility 1.1.3 artifacts. I'm not sure if this is an oversight in the constructed NOTICE or in the bin distribution tarball.

          Good catch, that inspired me to look into the mvn license:aggregate-add-third-party -X -e output from step1. I think we can run with -Dlicense.excludedScopes=test when generating.

          Attached patch 4 is based on the new run excluding test scope. Was able to take out a few test-only, not-bundled deps from patch 3.

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot for the review Sean Busbey ! The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012. jquery actually has been like this long ago... Looking at http://www.apache.org/dev/licensing-howto.html#permissive-deps , is the year required? I'm guessing the current way is written without year because the first 2 are 2005, 2013 and the last is 2012: ... hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-1.10.2.min.js hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery -------------------------------------------------------------------------------- Copyright jQuery Foundation and other contributors, https://jquery.org/ ... I'll be happy to update accordingly, but wanted to make sure - apache licensing seems is saying 'add a pointer', 'short note summarizing', and the example there didn't even mention copyright... There's no entry for the bundled Orca Logo from the HBase Server jar. It's mentioned in NOTICE, but LICENSE should have a complete reference for the CC-BY 3.0 license (found in LICENSE from the hbase-server-1.1.3.jar) Copied from there and added to hadoop LICENSE. I don't see any actual inclusion of HBase Shell 1.1.3, HBase IT Tests 1.1.3, or HBase Testing Utility 1.1.3 artifacts. I'm not sure if this is an oversight in the constructed NOTICE or in the bin distribution tarball. Good catch, that inspired me to look into the mvn license:aggregate-add-third-party -X -e output from step1. I think we can run with -Dlicense.excludedScopes=test when generating. Attached patch 4 is based on the new run excluding test scope. Was able to take out a few test-only, not-bundled deps from patch 3.
          Hide
          xiaochen Xiao Chen added a comment -

          Also attaching the updated scripts used, for reference.

          Show
          xiaochen Xiao Chen added a comment - Also attaching the updated scripts used, for reference.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          0m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13780
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845721/HADOOP-13780.04.patch
          Optional Tests asflicense
          uname Linux 2edde261e0b5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a605ff3
          modules C: . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11367/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 0m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13780 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845721/HADOOP-13780.04.patch Optional Tests asflicense uname Linux 2edde261e0b5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a605ff3 modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11367/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012.

          jquery actually has been like this long ago... Looking at http://www.apache.org/dev/licensing-howto.html#permissive-deps , is the year required?

          I'm guessing the current way is written without year because the first 2 are 2005, 2013 and the last is 2012:

          ...
          hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/jquery-1.10.2.min.js
          hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js
          hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery
          --------------------------------------------------------------------------------
          Copyright jQuery Foundation and other contributors, https://jquery.org/
          ...
          

          I'll be happy to update accordingly, but wanted to make sure - apache licensing seems is saying 'add a pointer', 'short note summarizing', and the example there didn't even mention copyright...

          The jQuery license specifically says the copyright notice has to be reproduced, so I'd presume that means the year is a relevant part of that reproduction. It's pretty easy to just list

          Copyright 2005, 2012, 2013 jQuery Foundation and other contributors, https://jquery.org

          Wether or not we include the copyright date, in v4 the jquery LICENSE section still needs to call out that there's a copy of 1.8.3 bundled in the hbase server jar.

          It looks like the hbase version changed from 1.1.3 in v3 to 1.2.4 in v4. I don't think there was any substantial LICENSE/NOTICE change between those versions, but I don't have time to confirm ATM. I don't think it's worth holding things up for that; I'll just file a follow-on if I find something.

          While reviewing the update for v4, I noticed there's an added blurb for a dependency that's BSD 4-clause. BSD 4-clause is the variant "with advertising clause" that's called out in the legal FAQ as not being category-a. It's not listed as any particular category, and isn't lised by the OSI. We can file a LEGAL asking if it's fine, but I suspect it isn't. Are we sure the version of JDOM we're using is BSD 4-clause? The current version of JDOM uses a one-off license that reads as cat-a to me (possibly calling for a NOTICE inclusion as well as LICENSE).

          Show
          busbey Sean Busbey added a comment - The jQuery entries look to cover things other than the bundling that's in the HBase Server 1.1.3 jar. Also the jQuery Foundation copyright entry is missing any year(s). The bit bundled in the HBase jar is version 1.8.3 with (c) 2012. jquery actually has been like this long ago... Looking at http://www.apache.org/dev/licensing-howto.html#permissive-deps , is the year required? I'm guessing the current way is written without year because the first 2 are 2005, 2013 and the last is 2012: ... hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/ static /jquery-1.10.2.min.js hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/jquery.js hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/ static /jquery -------------------------------------------------------------------------------- Copyright jQuery Foundation and other contributors, https: //jquery.org/ ... I'll be happy to update accordingly, but wanted to make sure - apache licensing seems is saying 'add a pointer', 'short note summarizing', and the example there didn't even mention copyright... The jQuery license specifically says the copyright notice has to be reproduced, so I'd presume that means the year is a relevant part of that reproduction. It's pretty easy to just list Copyright 2005, 2012, 2013 jQuery Foundation and other contributors, https://jquery.org Wether or not we include the copyright date, in v4 the jquery LICENSE section still needs to call out that there's a copy of 1.8.3 bundled in the hbase server jar. It looks like the hbase version changed from 1.1.3 in v3 to 1.2.4 in v4. I don't think there was any substantial LICENSE/NOTICE change between those versions, but I don't have time to confirm ATM. I don't think it's worth holding things up for that; I'll just file a follow-on if I find something. While reviewing the update for v4, I noticed there's an added blurb for a dependency that's BSD 4-clause. BSD 4-clause is the variant "with advertising clause" that's called out in the legal FAQ as not being category-a. It's not listed as any particular category, and isn't lised by the OSI. We can file a LEGAL asking if it's fine, but I suspect it isn't. Are we sure the version of JDOM we're using is BSD 4-clause? The current version of JDOM uses a one-off license that reads as cat-a to me (possibly calling for a NOTICE inclusion as well as LICENSE).
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the detailed explanations, Sean.

          jQuery

          Updated the copyright line to include the 3 years, and added a line for the v1.8.3 in hbase server.

          the hbase version changed from 1.1.3 in v3 to 1.2.4 in v4

          Yep, this is from YARN-5976 recently committed.

          JDOM

          You're correct, I was looking at https://github.com/hunterhacker/jdom/blob/jdom-1.1/core/LICENSE.txt and made it 4-clause BSD. But as you said with new additional text this should be considered a one-off license. So, updated it and also had a callout in NOTICE. (JDOM itself doesn't have a notice file, so followed the current style to point to its license+homepage.

          Also double checked other new deps' licenses are correct - this can be verified from the spreadsheet's 'Overrides' tab.

          Patch 5 attached to reflect the above.

          Show
          xiaochen Xiao Chen added a comment - Thanks for the detailed explanations, Sean. jQuery Updated the copyright line to include the 3 years, and added a line for the v1.8.3 in hbase server. the hbase version changed from 1.1.3 in v3 to 1.2.4 in v4 Yep, this is from YARN-5976 recently committed. JDOM You're correct, I was looking at https://github.com/hunterhacker/jdom/blob/jdom-1.1/core/LICENSE.txt and made it 4-clause BSD. But as you said with new additional text this should be considered a one-off license. So, updated it and also had a callout in NOTICE. (JDOM itself doesn't have a notice file, so followed the current style to point to its license+homepage. Also double checked other new deps' licenses are correct - this can be verified from the spreadsheet's 'Overrides' tab. Patch 5 attached to reflect the above.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 whitespace 0m 0s The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          0m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13780
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845834/HADOOP-13780.05.patch
          Optional Tests asflicense
          uname Linux a1fe20e9c787 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a55bd8
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11373/artifact/patchprocess/whitespace-eol.txt
          modules C: . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11373/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 whitespace 0m 0s The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 asflicense 0m 20s The patch does not generate ASF License warnings. 0m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13780 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845834/HADOOP-13780.05.patch Optional Tests asflicense uname Linux a1fe20e9c787 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a55bd8 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11373/artifact/patchprocess/whitespace-eol.txt modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11373/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Oops, patch 6 to fix whitespace from copying jdom license.

          Show
          xiaochen Xiao Chen added a comment - Oops, patch 6 to fix whitespace from copying jdom license.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          0m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13780
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845839/HADOOP-13780.06.patch
          Optional Tests asflicense
          uname Linux dec2a998a66f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a55bd8
          modules C: . U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11374/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 0m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13780 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845839/HADOOP-13780.06.patch Optional Tests asflicense uname Linux dec2a998a66f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a55bd8 modules C: . U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11374/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          +1 on v6. (I'd still prefer including the scripts, FWIW.)

          Show
          busbey Sean Busbey added a comment - +1 on v6. (I'd still prefer including the scripts, FWIW.)
          Hide
          andrew.wang Andrew Wang added a comment -

          Took a quick look at the v6 patch, if Sean's +1, I'm +1. Thanks for the reviews and continued work here!

          Show
          andrew.wang Andrew Wang added a comment - Took a quick look at the v6 patch, if Sean's +1, I'm +1. Thanks for the reviews and continued work here!
          Hide
          xiaochen Xiao Chen added a comment -

          Committed (the L&N only patch 6) to trunk. Thanks a lot Akira Ajisaka, Andrew Wang and Sean Busbey for the reviews and help!

          I'd still prefer including the scripts

          Out of HADOOP-12893 and this, I'm very eager to have the automation done. The scripts won't be lost. Will make sure the follow-on HADOOP-13948 is worked out so no one has to play lawyer for alpha3.

          Show
          xiaochen Xiao Chen added a comment - Committed (the L&N only patch 6) to trunk. Thanks a lot Akira Ajisaka , Andrew Wang and Sean Busbey for the reviews and help! I'd still prefer including the scripts Out of HADOOP-12893 and this, I'm very eager to have the automation done. The scripts won't be lost. Will make sure the follow-on HADOOP-13948 is worked out so no one has to play lawyer for alpha3.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11077 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11077/)
          HADOOP-13780. LICENSE/NOTICE are out of date for source artifacts. (xiao: rev 8850c056a5af774e3c614b57135709baed718c7a)

          • (edit) NOTICE.txt
          • (edit) LICENSE.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11077 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11077/ ) HADOOP-13780 . LICENSE/NOTICE are out of date for source artifacts. (xiao: rev 8850c056a5af774e3c614b57135709baed718c7a) (edit) NOTICE.txt (edit) LICENSE.txt

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              busbey Sean Busbey
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development